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Managementuittreksel 

TNO  1  Kennis  voor  zaken 

Analyse  methoden  en  modellen  voor  kleine 
uitgestegen  eenheden 


Dil  onderzoek  gaat  na  in  hoeverre  TNO  beschikt  over  n^thoden  en 
modellen  voor  de  analyse  van  operationele  effectiviteit  van  kleine 
uitgestegen  eenheden.  Dit  inzicht  wordt  in  andere  studies  gebruikt  om  aan  te 
geven  vvelke  op  welke  wijze  operationele  effectiviteit  kan  worden 
gekwalificeerd  en  gekwantificeerd. 


Probleemstelling 

Het  (Joel  van  het  project  ‘Meelmethoden’ 
binnen  het  prograrruna  Optreden  Kleine 
Eenheden  is  het  inzichielijk  maken  voor  de 
Nederlandse  krijgsmacht  van  methoden  die 
prakiisch  toepasbaar  zijn  voor  het  meten  en 
beoordelen  van  de  inzei  en  het  effect  van 
een  optreden  van  een  kleine  eenheid. 


Beschrijving  van  de 
werkzaamhedcn 

Er  is  gestart  met  een  inventarisatie  van 
beschikbare  meelmethoden  en  modellen 
binnen  TNO  Defensie  en  Veiligheid.  Door 
gebruik  te  maken  van  de  eerder  vastgesielde 
lijst  met  taken  van  een  kleine  eenheid  in  het 
project  ‘Small  unit  tactics  in  peace  keeping 
operations’,  is  per  beschikbaar  model  en 
methode  vastgesteld  in  hoeverre  deze 


toepasbaar  en  relevant  is  om  iets  over  de 
uitvoering  en/of  de  opbrengst  van  deze 
taken  vast  te  siellen. 

Resultaten  en  conciusies 

De  srudie  laat  zien  dat,  hoewel  TNO 
beschikt  over  een  reeks  van  modellen  en 
methodieken,  niei  alle  taken  van  een  kleine 
eenheid  kunnen  worden  beoordeeld  (op 
onder  andere  hun  effectiviteit).  Het  zijn 
vooral  de  taken  in  het  lage  geweldspectrum 
waarin  de  methodieken  en  modellen  (nog) 
met  zijn  toegepasl  en  getoetst  op  hun 
bruikbaarheid.  Een  aantal  van  deze 
modellen  en  methodieken  is  potent ieel  wel 
bruikbaar. 

Toepasbaarheid 

De  resultaten  van  deze  studie  dragen  bij  aan 
het  maken  van  een  overzicht  van  de  kennis, 
kunde  en  inffastructuur  ten  behoeve  van  het 
programma  Optreden  Kleine  Eenheden. 

De  inzichten  van  deze  studie  worden 
gebruikt  in  een  case-stud ie  van  het 
programma  OKE  waarin  op  een 
aanschouwelijke  manier  wordt  zichtbaar 
gemaakt  hoe  het  optreden  van  een  kleine 
eenheid  kan  worden  geo  puma  Li  see  rd  en  dat 
deze  effeciiviteitverbetering  ook 
gekwantificeerd  kan  worden. 
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Samenvatting 


De  doelstelling  die  ten  grondslag  ligt  aan  het  onderzoeksprogramma  Optreden  Kleine 
Eenheden  (OKE)  is  dat  de  Nederlandse  Krijgsmacht  moet  kunnen  beschikken  over 
adequaat  ingerichte  en  uitgeruste  eenheden  die  taken  in  het  gehele  scala  aan 
vredesondersteunende  operaties,  met  een  wisselend  geweldsniveau,  aantoonbaar 
doeltreffend  en  doelmatig  moeten  kunnen  uitvoeren,  Bij  de  uitvoering  van 
vredesondersteunende  operaties  ligt  de  nadruk  op  het  optreden  van  kleinere  verbanden. 
Om  de  KL  ook  in  de  toekomst  te  kunnen  blijven  ondersteunen  bij  materieel-  en 
beleidsvraagstukken  op  het  gebied  van  OKE,  is  een  inventarisatie  gemaakt  van  de 
momenteel  beschikbare  onderzoeksmethoden  en  -modellen  die  ingezet  worden  om 
vraagstukken  betreffende  het  optreden  en  de  operationele  effectiviteit  te  beantwoorden. 
Om  na  te  gaan  of  de  beschikbare  modellen  en  methoden  geschikt  zijn  om 
domeinspecifieke  (Kleine  Eenheid)  vraagstellingen  te  kunnen  onderzoeken,  is  de 
takenlijst  uit  het  project  ‘Tactisch  Optreden  kleine  eenheden’  als  uitgangspunt 
genomen.  Er  is  voor  elke  specifieke  taak  gekeken  in  welke  mate  de  modellen  bruikbaar 
zijn.  Door  de  modellen  te  kwalificeren  op  basis  van  directe  toepasbaarheid  zijn  drie 
categorieen  ontstaan,  namelijk  praktisch  toepasbaar  (alle  onderliggende  relaties  zijn 
bekend),  theoretisch  mogelijk  (er  wordt  een  causaal  verband  verondersteld  tussen  twee 
of  meerdere  grootheden  maar  de  relaties  zijn  nog  niet  of  nauwelijks  gekwantificeerd)  en 
logisch  mogelijk  (er  is  een  verband  mogelijk  tussen  twee  gemeten  grootheden  maar  het 
is  nog  niet  duidelijk  hoe  dat  verband  ligt),  Aan  de  hand  van  een  matrix  van  taken  x 
methoden  is  nagegaan  in  hoeverre  de  bij  TNO  beschikbare  methoden/modellen 
gekwalificeerd  zijn  om  uitspraken  te  doen  over  de  resultaten  van  taken. 

Bij  analyse  van  de  gegevens  blijkt  dat  geen  enkel  methode  uitspraken  doet  over  alle 
taken  van  de  kleine  eenheid.  Bovendien  valt  op  dat  TNO  weliswaar  beschikt  over 
meerdere  modellen  die  uitspraken  doen  over  specifieke  taken  ,  maar  deze  taken  vallen 
alien  in  de  categoric  ‘hoger  geweldspectrum’.  TNO  beschikt  niet  over  expliciete 
modellen  en  methoden  die  praktisch  toepasbaar  zijn  op  taken  die  met  name  vallen  in  de 
categoric  ‘niet  gevechtstaken’. 

Een  klein  aantal  methoden  en  modellen  die  bruikbaar  zijn  gebleken  voor  de 
gevechtstaken  hebben  ook  het  potentieel  om  te  worden  toegepast  voor  niet- 
gevechtstaken. 
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Summary 


One  of  the  primary  goals  of  the  Netherlands  Armed  Forces  is  to  have  at  their  disposal 
units  that  are  adequately  equipped  and  outfitted  to  carry  out  tasks  across  the  complete 
range  of  peace-supporting  missions,  varying  in  levels  of  hostility.  To  this  end  a 
scientific  research  program  was  formulated  which  encompasses  a  number  of  projects 
which  focus  on  various  aspects  of  operations  of  teams  of  dismounted  soldiers.  The  aim 
of  this  project  is  to  identify  scientific  methods  and  models  which  facilitate  the 
measurement  of  operational  effectiveness  of  dismounted  small  units. 

Measuring  operational  effectiveness  of  dismounted  small  units  requires  the  use  of 
methods  models  and  simulations.  Within  this  project  we  have  made  an  inventory  of  the 
available  methods  and  models  within  the  TNO  technology  portfolio.  In  order  to  be  able 
to  asses  the  usefulness  of  these  methods  and  models  for  the  tasks  the  dismounted  small 
units  are  confronted  with,  the  task  decomposition  developed  in  the  project  Small  unit 
tactics  in  peace  keeping  operations’  (Smeenk  et  al.,  2004)  was  used  as  a  guideline.  The 
usefulness  of  the  methods  and  models  was  estabhshed  by  classifying  these  into  three 
categories,  i.e.  logically  possible,  theoretically  feasible  and  practically  applicable.  For 
each  of  the  tasks  the  models  and  methods  were  reviewed  with  respect  to  the  mentioned 
categories.  This  resulted  in  one  matrix  of  tasks  and  models/methods. 

The  analysis  of  the  matrix  suggests  that  not  one  single  method/model  is  practically 
applicable  to  all  the  identified  tasks  of  the  dismounted  small  unit.  Furthermore, 
although  TNO  has  access  to  a  range  of  models/methods  which  are  practically  applicable 
to  a  number  of  tasks,  these  tasks  can  al  be  considered  as  high  intensity  tasks.  TNO  does 
not  have  access  to  methods/models  which  are  explicitly  and  practical  applicable  in  the 
low  intensity  range  of  the  task  spectrum. 

A  small  number  of  models  seem  to  have  the  potential  to  fill  the  identified  gap. 
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1  Introduction 


One  of  the  primary  goals  of  the  Netherlands  Armed  Forces  is  to  have  at  their  disposal 
units  that  are  adequately  equipped  and  outfitted  to  carry  out  tasks  across  the  complete 
range  of  peace-supporting  missions,  varying  in  levels  of  hostility.  Key  to  this  task 
execution  is  that  the  results  of  the  operations  should  be  discernibly  effective  and 
appropriate.  In  the  execution  of  peace-supporting  operations,  the  emphasis  is  on 
operating  in  smaller  groups.  This  form  of  operating,  under  often  difficult  circumstances, 
in  complex  terrain  and  with  a  whole  range  of  new  threats,  makes  new  demands  on  the 
composition  of  such  units  and  the  kit  with  which  they  are  outfitted. 

The  defence  research  program  entitled  Small  Unit  Operations  generates  knowledge, 
both  theoretical  and  applicable,  and  infrastructure  in  order  to  quantify  and  qualify  the 
operational  effectiveness  (OE)  of  small  units  during  various  types  of  peace-supporting 
operations.  The  generated  knowledge  and  infrastructure  will  provide  TNO  with  the 
tools  needed  to  respond  more  adequately  and  more  efficiently  to  future  questions 
regarding  small  units. 

In  the  generation  of  the  knowledge  needed  to  respond  to  operational  questions 
regarding  small  units,  two  issues  are  of  foremost  importance:  1)  the  factors  indicative  of 
the  performance  of  small  units  (think  of  outfitting,  unit  composition,  and  situational 
factors)  and  2)  methods  to  measure  or  assess  the  value  of  the  various  factors.  Once  the 
factors  are  specified  and  the  value  is  known,  it  is  possible  to  model  the  performance  of 
small  units  and,  from  there,  to  acquire  predictive  insight  into  how  changes  in  the 
identified  factors  affect  a  unit’s  performance. 

The  present  study  aims  first  to  identify  methods  available  to  TNO  and  the  Netherlands 
Armed  Forces  with  which  to  measure  or  assess  the  operational  effectiveness  of  small 
units,  and  the  factors  that  determine  OE.  Once  this  has  been  achieved,  the  second  aim  of 
tliis  study  is  to  take  the  first  step  toward  the  identification  of  the  determinants  of  OE  for 
small  units.  That  is,  which  factors  determine  the  OE  of  a  small  unit  and  how  do  the 
factors  relate  to  each  other? 

This  report  is  an  account  of  the  activities  carried  out  in  order  to  reach  the  desired  project 
goals.  In  Chapter  2  we  examine  various  general  premises  upon  which  the  work  was 
based.  We  offer  a  conceptual  model  and  define  various  key  concepts.  In  Chapter  3  we 
make  an  inventory  and  evaluation  of  the  empirical  methods  available  with  which  to 
measure  and  assess  OE  and  its  determinants.  In  Chapter  4  we  inventory  various  models 
and  simulations  relevant  for  assessing  the  OE  of  small  units.  In  Chapter  5  we  make  first 
tentative  step  towards  identifying  the  contents  of  a  model  representing  the  determinants 
of  OE  for  small  units.  Chapter  5  recaps  the  main  conclusions  from  the  study. 
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2  Methods 


2.1  Operational  effectiveness 

Measuring  Operational  Effectiveness  of  small  dismounted  units:  Methods  and  Models 
Before  presenting  the  results  of  the  inventory  of  research  methods  and  models  that  can 
be  used  to  measure  or  assess  the  operational  effectiveness  of  an  operation  of  a  small 
unit,  we  need  to  first  define  operational  effectiveness  and  how  it  can  be  estimated.  In 
the  project  Measures  of  effectiveness  for  the  RNLA  (2002)  in  the  small  unit  operations 
research  program,  OE  was  described  as  the  end  result  of  the  task  execution  of  a  small 
unit  with  a  particular  make  up  and  kit  in  a  particular  environment.  OE  describes  the 
degree  to  which  the  achieved  effects  match  the  desired  effects.  Essential  to  an  accurate 
assessment  of  operational  effectiveness  is  a  specific  definition  of  what  the  desired 
effects  or  state  are.  In  Effects  and  Values,  we  described  in  considerable  detail  that  it  is 
necessary  to  define  realization  criteria,  which  must  be  met  in  order  to  conclude  that  an 
operation  was  effective.  Because  in  peace-supporting  missions  the  achievement  of  goals 
is  not  always  dichotomous,  the  realization  criteria  are  ideally  not  discrete  values,  but  are 
based  on  continua,  which  can  take  on  a  whole  range  of  values.  Using  these  values,  it  is 
possible  to  specify  when  the  desired  effects  have  been  completely  achieved,  achieved  to 
a  satisfactory  level,  and  when  the  levels  are  still  unsatisfactory. 

In  the  present  study,  we  examined  ways  to  assess  the  levels  of  key  variables  and 
integrate  these  assessments  in  order  to  draw  conclusions  regarding  the  OE.  This 
conceptual  process  is  illustrated  in  Figure  1  and  is  as  follows.  Operational  effectiveness, 
preferably  as  a  single  value,  is  a  combined  measure  based  on  the  different  variables  that 
can  affect  the  performance  of  a  small  unit.  These  variables  can  be,  for  example, 
characteristics  of  the  small  unit  itself  (composition,  size,  physical  state),  the 
environment  in  which  the  unit  must  operate  (climate,  landscape,  culture),  or 
organizational  constraints  (orders,  infrastructure,  kit).  In  order  to  calculate  the  OE  based 
on  the  relevant  variables,  models  can  be  developed  mathematically  in  which  the 
variables  are  specified  as  well  as  how  the  variables  relate  to  each  other.  Though  it  is 
possible  to  theoretically  identify  variables  that  determine  OE,  in  order  to  calculate  the 
operational  effectiveness  itself,  it  is  necessary  to  assign  values  to  the  variables.  These 
input  values,  however,  do  not  appear  out  of  thin  air,  but  must  be  measured  empirically. 
In  some  cases  this  is  as  simple  as  measuring  the  temperature  with  a  thermometer.  In 
cases,  however,  in  which  the  variables  are  related  to  more  behavioural  aspects  of  an 
operation  (e.g.,  winning  the  hearts  and  minds  of  a  local  population),  this  is  not  so 
simple,  and  more  complex  data  collection  methods  are  called  for. 
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Figure  1  EtTecl  Based  Operations  -  framework. 

In  practice,  Figure  1  is  useful  when  trying  to  estimate  the  effects  of  a  particular  aspect 
of  a  future  mission.  For  example,  using  the  above  model,  we  can  gain  a  better 
understanding  of  the  effects  of  a  particular  piece  of  kit  on  the  operational  effectiveness. 
We  can  measure  what  the  effect  of  the  kit  is  on,  say,  the  situational  awareness,  and  then 
calculate  mathematically  what  the  effects  of  the  resulting  change  in  situational 
awareness  are  on  the  total  operational  effectiveness. 

The  other  side  of  the  measurement  of  operational  awareness  is  the  evaluation  of  the 
effectiveness  of  an  operation  that  has  actually  taken  place.  For  example,  one  may  want 
to  know  the  effects  of  the  activities  carried  out  by  a  small  unit  to  help  rebuild  the 
infrastructure  of  a  village.  In  this  case,  the  process  described  above  does  not  suffice. 
Instead,  the  link  between  the  OF  and  empirical  data  is  more  direct  and  is  represented  in 
Figure  2.  On  the  left,  we  see  the  realization  criteria  that  must  be  met  in  order  to 
conclude  that  an  operation  was  completely,  partially  or  not  at  all  effective.  In  order  to 
ascertain  the  degree  to  which  the  criteria  are  met,  empirical  methods  of  data  collection 
can  be  used.  The  results  of  these  empirical  measurements  are  then  compared  to  the 
realization  criteria.  The  discrepancy  between  the  measured  values  and  the  desired 
values  of  the  different  variables  can  be  integrated  in  order  to  draw  conclusions 
regarding  the  resulting  operational  effectiveness. 
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Intent  =  I  tasks 


Figure  2  Etfecl  Based  Operations  and  models. 


Though  conceptually,  the  process  of  assessing  the  operational  effectiveness  of  a  small 
unit  operation,  there  are  important  aspects  of  measurement  and  pre-estimation  that 
determine  the  quahty  of  the  evaluation  and  its  applicability. 

2.2  Procedure 

In  order  to  give  an  overview  of  the  methods  TNO  possesses  to  determine  operational 
effectiveness,  a  List  of  tasks,  their  objectives  and  results  was  abstracted  from  the  project 
‘small  unit  tactics  in  peace  keeping  operations’  (Smeenk  et  al,  2004).  These  tasks  were 
used  as  the  basis  for  this  project  and  sunmiarized  in  Table  1.  The  translation  of  the 
tasks,  their  goals  and  results  can  be  found  in  Appendix  A.  The  usefulness  of  all 
available  research  and  data  collection  methods  available  at  TNO  Defence,  Safety  and 
Security,  was  determined  and  categorized.  Categorization  was  done  based  on  the 
maturity  and  applicability  of  a  specific  method  in  combination  with  the  selected  task 
within  TNO  Defence  Safety  and  Security.  Three  categories  were  distinguished;  lx)gical, 
Theoretical,  and  Practical.  The  description  and  criteria  for  these  three  categories  are 
listed  below. 
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Table  I  Tasks  exiracied  from  'Small  uait  tactics  in  peace  keeping  operations’  (Smeenk  el  al.,  2004). 


Peace-keeping  tasks 

Combat  Tasks 

Construct  observation  posts 

Combat  patrol 

Man  observation  posts 

Take  a  position 

Conduct  patrol 

Conduct  a  fire  raid 

Security  patrol 

Conduct  a  mounted  attack 

Contact-patrol 

Conduct  a  counter  attack 

Patrol  to  locate  mines  and  explosives 

Conduct  a  counter  strike 

Social  Patrol 

Establish  reinforcements 

Inspect  vehicles  and  persons/man  checkpoints 

Establish  a  fire  base 

Inspect  sites,  areas  and  buildings 

Establish  a  team  collection  area 

Cordon  and  search  sites,  areas  and  buildings 

Perform  a  tactical  movement 

Monitor/protect  objects,  areas  and  persons 

Perform  an  infiltration  or  exfiltration 

Escort/convoy  security  and  protection 

Conduct  an  attack  on  foot  or  a  dismounted 
attack 

Quick  reaction  force 

Weapon  collection  activities 

Conduct  liaison 

Showing  the  force/flag  (e.g.  shooting 
exercises) 

Spreading  of  Info  Ops  products  (e.g.  flyers) 
Inform  civilians  and  local  organisations  about 
mine  awareness 

Reconnaissance  with  other  means  than  patrols 
(RPV,  EOV,  Radar,  etc.) 

Provide  assistance  in  rebuilding  the  local 
infrastructure 

Supply  class  1,  11,  III,  IV,  V 

Re-establishing  communications 

Transport  support  (drivers,  vehicles) 

Support  to  crowd  and  Riot  Control  (CRC) 
operations 

Support  to  arrests  (security  and  protection) 
Transport  support  (drivers,  vehicles) 

Conduct  an  advance 

2.2.1  Logical 

At  the  highest  abstraction  level,  methods  can  be  identified  which  are  plausible  and 
could  be  executed  but  would  require  extensive  research  in  identifying  the  relevant 
factors  and  their  causal  relationships.  Furthermore  parameters  and  weights  will  not  be 
readily  available  and  would  require  extensive  empirical  research. 

2.2.2  Theoretical 

Less  abstract  than  logical  methods,  theoretically  feasible  methods  do  not  require  the 
identification  of  factors  and  their  causal  relationships.  Models  and  methods  which  are 
best  described  by  this  qualification  are  relevant  and  have  been  tested  but  do  not  include 
the  domain  specific  parameters  and  weights.  These  models,  to  be  applicable,  still 
require  considerable  empirical  testing  and  the  gathering  of  domain  specific  parameters 
and  weights. 
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2.2J  Practical 

Models  and  methods  which  are  best  qualified  as  being  practical  have  been  empirically 
tested  but  also  in  the  relevant  domain.  These  models  and  methods  can  be  applied 
without  much  extra  effort. 

Common  scientific  research  methods  and  data  collection  methods  include  experiments, 
field  studies,  archival  research,  interviews,  observations,  surveys  and  the  analysis  of 
pre-existing  data.  These  methods  are  described  in  great  detail  in  Appendix  0, 

These  methods  are  being  used  to  gather  input  (data)  for  models  as  explained  in 
Figure  2.  The  ‘throughput’  consists  of  a  scheme  of  comparisons  and  formulae  (model) 
that  describe  the  relationship  between  empirically  obtained  measures  (input)  and  the 
result  for  a  specific  operational  context  (output). 

The  models  that  can  be  used  to  understand  and  describe  the  relationship  between  input 
and  output  are  described  in  great  detail  in  Appendix  0.  These  models  were  reviewed  for 
the  tasks,  their  objectives  and  results  from  ‘small  unit  tactics  in  peace  keeping 
operations’  (Smeenk  2004)  and  categorized  as  logical,  theoretical  or  practical.  The 
complete  matrix  can  be  found  in  Appendix  0. 
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3  Empirical  methods 


The  key  to  answering  any  research  question  is  to  choose  the  right  method  with  which  to 
find  the  answer.  Just  as  you  would  not  use  a  thesaurus  to  find  a  recipe  for  Beef 
Wellington,  you  should  not  use  an  experiment  to  study  a  phenomenon  that  would  be 
better  suited  to  research  using  interviews.  In  this  chapter  we  inventory  various  methods 
with  which  empirical  data  can  be  collected  that  can  later  be  used  as  input  for  a  more 
integrative  model  or  simulation  with  which  complex  concepts  such  as  operational 
effectiveness  can  be  measured  or  assessed. 

3.1  Longitudinal  and  cross-sectional  research 

In  conducting  research  it  is  often  desirable  to  determine  causality.  That  is  to  say,  to  be 
able  to  conclude  that  one  variable  or  process  causes  another.  This  is  often  very  difficult, 
particularly  with  social  or  behavioural  phenomena.  Other  times  it  is  desirable  to  contrast 
and  compare  two  or  more  situations  in  order  to  determine  which  is  better  or  more 
effective.  In  order  to  aid  in  these  processes  of  the  determination  of  causality  and 
multiple  comparisons,  there  are  two  main  ways  in  which  a  study  can  be  designed  and 
administered:  longitudinal  and  cross  sectional. 

3J.I  Longitudinal  designs 

Longitudinal  refers  to  research  in  which  a  particular  measure  is  admiaistered  repeatedly 
over  time.  Census  studies  are  examples  of  longitudinal  designs:  censuses  may  be  taken 
once  every  so  many  years  in  order  to  determine  the  make-up  of  the  population  on  a 
variety  of  dimensions  such  as  age,  occupations,  household  composition,  etc.  Using  a 
longitudinal  design  also  allows  causality  to  be  determined.  The  dimensions  of  interest 
are  measured  in  order  to  establish  a  baseline  (to;  the  number  of  firearms  turned  into 
occupying  forces  over  a  period  of,  for  example,  a  week),  a  manipulation  of  the  relevant 
variable  is  introduced  (a  PsyOps  money-for-firearms  campaign)  and  the  dimension  of 
interest  is  measured  a  second  time  (tf,  the  number  of  firearms  turned  in  during  the  first 
week  of  the  campaign).  By  comparing  to  and  ti,  we  can  determine  the  increase  in  the 
number  of  firearms  turned  in  as  a  direct  result  of  the  PsyOps  campaign  (the  PsyOps 
campaign  ‘causes’  people  to  turn  in  their  weapons). 

We  can  also  use  longitudinal  designs  to  compare  multiple  situations.  These  types  of 
designs  are  usually  referred  to  as  repeated  measures.  For  example,  if  we  want  to  test 
soldiers’  ability  to  navigate  using  traditional  locating  and  navigational  devices  versus 
using  Soldier  Digital  Assistants  (SDA),  we  may  ask  a  group  of  soldiers  to  perform 
navigational  task  first  using  traditional  equipment  and  then  a  second  time  using  SDA’s. 
The  main  drawback  of  using  longitudinal  designs  is  that  it  is  difficult  to  determine  if  the 
results  that  are  found  are  due  to  the  interventions  or  due  to  the  passage  of  time.  For 
example,  how  do  we  know  that  people  wouldn’t  have  started  turning  in  more  firearms 
anyway,  even  without  the  introduction  of  the  PsyOps  campaign?  Maybe  there  was 
something  else  that  happened  that  encourage  people  to  start  turning  in  their  weapons, 
for  example,  a  revered  civic  leader  made  a  statement  that  he  was  in  favour  of  the 
disarming  of  civilians.  To  tease  apart  the  effects  of  our  intervention  from  other  effects 
in  the  environment,  v/e  need  a  different  type  of  design:  the  cross-sectional  design. 
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3.1.2  Cross-sectional  designs 

Cross-sectional  designs  compare  two  groups  who  are  similar  on  a  number  of  relevant 
variables,  using  a  measure  administered  at  one  moment  in  time.  This  is  the  classic 
design  using  control  groups.  For  example,  we  want  to  know  the  effect  of  a  particular 
medication  on  relieving  symptoms  of  the  common  cold.  To  test  this  we  can  give  one 
group  of  people  with  a  cold  the  medication  (treatment  group)  and  the  other  groups 
nothing  (or,  better  yet,  a  placebo;  control  group).  To  test  the  effectiveness  of  the 
medication,  we  can  ask  the  participants  to  tell  us  how  they  fell  on  a  number  of  cold- 
related  dimensions  (sneezing,  stuffy  nose,  headache,  sore  throat,  coughing).  If  the 
treatment  group  indicates  significantly  fewer  symptoms  than  the  control  group,  we  can 
conclude  that  the  medication  ‘caused’  people  to  feel  better. 

It  is  possible  to  combine  cross-sectional  designs  with  longitudinal  designs  in  order  to 
gain  more  insight  in  to  the  phenomenon  of  interest.  To  continue  with  the  example  of  the 
money-for-firearms  campaign:  we  choose  two  comparable  villages.  Over  a  period  of  a 
week  we  count  the  number  of  firearms  that  villagers  spontaneously  turn  over  to  the 
occupying  forces.  We  subsequently  start  a  money-for-firearms  campaign  in  one  of  the 
villages,  but  not  in  the  other  (An  additional  option  would  be  to  compare  a  money-for- 
firearms  campaign  with  a  campaign  encouraging  villagers  to  turn  in  firearms  without 
the  promise  of  a  reward).  We  measure  the  number  of  firearms  turned  in  both  villages  in 
the  first  week  of  the  campaign.  In  principle,  we  hope  for  the  following  pattern  of 
results:  in  the  first  measurement  week  the  inhabitants  of  both  villages  turn  in  equal 
numbers  of  firearms;  in  the  second  measurement  week  inhabitants  of  the  village  that  has 
the  money-for-firearms  campaign  turn  in  more  weapons  than  the  inhabitants  of  the 
control  village.  Assuming  that  people  from  elsewhere  did  not  turn  in  their  firearms  in 
the  treatment  village,  we  can  conclude  that  the  money-for-firearms  campaign  ‘caused’ 
people  to  turn  in  their  firearms. 

Alternatively,  if  we  find  that  in  the  second  measurement  week  equal  numbers  of 
firearms  were  turned  in  both  villages,  we  have  reason  to  believe  that  something  else  is 
going  on.  If,  in  the  second  measurement  week  both  villages  differ  from  the  baseline 
measure,  but  not  from  each  other,  we  can  conclude  that  there  was  some  other  factor  that 
occurred  between  the  first  and  second  measurement  weeks  that  is  responsible  for  the 
increase.  On  the  other  hand,  if  neither  village  shows  a  change  between  the  two 
measurement  weeks,  we  can  conclude  that  the  PsyOps  campaign  was  not  effective. 

3.2  Reliability 

The  rehability  of  a  measurement  refers  to  the  accuracy  of  the  measure  used.  Reliability 
is  usually  assessed  by  measuring  the  same  object  or  dimension  more  than  once  (using 
either  the  same  or  a  similar  instrument).  If  the  two  measurement  moments  yield  results 
identical  within  a  specified  margin  of  error  (determined  by  correlating  the  different 
scores),  the  measure  is  considered  to  be  reliable.  For  example,  the  Stanford-Binet  IQ- 
test  has  a  standard  error  of  3  points.  This  means  that  if  Person  X  takes  the  test  on 
Monday  and  then  again  on  Friday,  his  two  IQ-scores  can  differ  by  plus  or  minus 
3  points.  This  test  is  considered  reliable  because  it  accurately  measures  something  we 
define  as  IQ;  that  is,  multiple  test  administrations  yield  similar  IQ-scores  (whether  what 
the  lest  measure  is  really  IQ  or  not  is  a  question  of  validity,  not  reliability,  and  will  be 
addressed  in  the  next  section).  An  IQ-test  with  a  standard  error  of,  say,  25  points  would 
be  less  accurate  and  therefore  less  reliable. 
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3.3  Validity 

Validity  in  measurement  is  that  the  method  used  actually  measures  what  it  is  supposed 
to  measure.  Though  this  would  seem  to  speak  for  itself,  validity  is  not  always  easy  to 
achieve.  Let’s  say  we  wanted  to  know  the  effect  of  a  small  unit’s  help  at  repairing  a 
school  on  the  local  population’s  attitude  toward  the  occupying  force.  To  find  out  we 
could  ask  the  locals  how  happy  they  are  with  the  school.  The  assumption  here  is  that  if 
the  small  unit  does  something  that  makes  the  locals  happy,  their  attitudes  will  become 
more  positive  towards  the  occupying  forces.  However,  this  question  may  or  may  not  be 
a  valid  measure  of  the  locals’  attitudes  on  this  dimension:  locals  may  be  very  happy  that 
their  school  has  been  repaired,  but  they  may  still  resent  the  foreign  presence  in  their 
village. 

Validity  is  difficult  to  determine  because  the  only  true  way  to  evaluate  validity  is  to 
compare  a  particular  measure  to  one  that  is  know  to  be  valid.  Direct  measures  often 
have  more  vahdity  than  indirect  ones  because  it  is  easier  to  rule  out  alternative 
explanations.  We  could,  for  example,  ask  locals  if  they  approve  of  the  presence  of  the 
occupying  forces  in  their  village.  This  question  is  a  more  direct  measure  of  what  we 
want  to  know.  The  validity  is  therefore  likely  to  be  higher  than  the  school  question.  To 
find  out  the  validity  of  the  school  question,  we  could  correlate  locals’  responses  to  the 
two  questions.  If  the  correlation  is  positive  and  high,  the  school  question  is  valid.  We 
could  also  correlate  the  direct  question  with  other  measures:  attacks  on  foreign  soldiers, 
threats  or  insults  shouted  towards  foreign  soldiers  etc.  If  the  correlation  is  negative  and 
high,  we  can  conclude  that  all  are  valid  measures  of  the  locals’  attitudes  towards  the 
occupying  forces. 

The  vahdity  of  a  measure  also  depends  on  the  construct  being  measured.  Measuring  the 
length  of  a  house  is  easy  because  the  construct  'length’  is  relatively  straightforw^ard  in 
this  case.  In  the  section  on  reliability  we  gave  the  example  of  IQ.  IQ  is  difficult  to 
measure  in  a  vahd  manner  because  the  construct  TQ’  is  very  complex  and  there  is  much 
disagreement  on  what  makes  a  person  intelUgent.  For  this  reason,  it  is  difficult  to 
determine  the  validity  of  IQ-tests. 

In  general  terms,  we  can  distinguish  between  two  important  types  of  validity: 

Internal  validity  which  is  the  degree  to  which  differences  in  performance  can  be 
attributed  unambiguously  to  an  effect  of  an  independent  variable,  as  opposed  to  an 
effect  of  another  (uncontrolled)  variable;  an  internally  valid  study  is  free  of 
confounders. 

External  validity  which  is  the  extent  to  which  the  results  of  a  research  study  can  be 
generalized  to  different  populations,  settings  and  conditions. 

3.4  Shelf  life 

Once  data  have  been  collected  on  a  particular  topic  of  study,  they  can  be  brought  in 
connection  with  other  data  in  a  model  or  generahzed  to  other  situations.  However,  the 
validity  of  extrapolating  the  meaning  of  data  through  such  activities  is  not  always  the 
same:  data  have  a  shelf  life.  The  circumstance  in  which  data  are  collected  can  result  in 
the  data  only  being  meaningful  in  those  circumstances. 

Consider  measuring  the  boiling  point  of  water  a  sea  level.  Water  boils  at  100  at  sea 
level.  However,  this  measurement  does  not  generalize  to  other  altitudes:  the  boihng 
point  of  water  is  negatively  correlated  with  altitude,  that  is,  the  higher  you  are,  the 
lower  the  temperature  at  which  water  boils.  Consequently,  if  you  apply  the 
measurements  taken  at  sea  level  to  a  situation  occurring  at  a  higher  altitude,  your 
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extrapolation  will  yield  incorrect  results.  In  another  example,  the  results  of  studies  of 
socially  correct  behaviour  in  Afghanistan  are  not  automatically  applicable  in  Iraq.  In 
sum,  data  have  a  shelf  life  in  terms  of  space. 

Data  also  have  a  shelf  life  in  terms  of  time.  Though  the  chances  are  good  that  water  at 
sea  level  will  always  boil  at  100  °C,  measurements  of  more  behavioural  or  cultural 
variables  will  change  over  time.  Cultural  norms  change,  people’s  attitudes  change,  and 
available  resources  change.  As  a  result,  measuring  people’s  attitudes  towards  occupying 
forces  today  does  not  necessarily  predict  their  attitudes  a  year  from  now.  Consider  the 
change  in  Europeans’  attitudes  towards  the  United  States  in  the  past  five  years. 
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4  Data 


4.1  The  matrix 

As  explained  in  Chapter  2.2,  tasks  were  abstracted  frona  ‘small  unit  tactics  in  peace 
keeping  operations’  (Smeenk  et  al,  2004).  The  tasks  were  placed  in  a  matrix,  together 
with  the  available  models,  as  shown  in  Table  3.  The  available  models  are  summarized 
in  Table  2  (Jaiswa  1997), 


Table  2 


Available  models  and  ilieir  usability. 


Model 


Description  (usability  of  methods) 


Search  and  damage  assessment 

Simulation  of  military  systems 
War  games 


Homogeneous  and  heterogeneous 
combat  models 


Cost  effectiveness  analysis 


Multiple  Criteria  Analysis 


Analytic  optimization  models 


Heuristic  optimization  models 


Drama  theory 


Quantitative  threat  assessment: 
static  and  dynamic  analyses 
Belief  networks  and  influence 
diagrams 


Search  models  can  increase  the  situation  awareness  or  can  give  information  about 
the  position  of  an  enemy.  The  output  of  damage  assessment  is  the  probability  or  the 
proportion  of  the  area  of  a  target  damaged  by  a  weapon  or  weapons.  So  these 
methods  can  be  used  for  every  task  where  the  yield  is  to  give  information  about  the 
environment,  the  enemy  or  the  consequences  of  firing  a  weapon. 

The  yields  of  all  tasks  can  be  analysed  with  simulation.  Simulation  is  like  representing 
the  truth  in  a  model,  so  everything  in  reality  can  be  analysed  through  simulation. 

War  games  are  simulations  of  a  military  operation  involving  two  or  more  opposing 
forces.  War  games  can  be  used  where  information  is  needed  for  training  capabilities, 
the  effectiveness  of  a  procedure  or  plan  or  doctrines  or  weapon  systems. 
Homogeneous  models  are  based  on  attrition  warfare.  The  output  of  these  models  is 
which  party  will  probably  win  the  conflict.  So  these  models  can  be  used  when 
information  has  to  be  given  about  the  effectiveness  of  a  combat,  such  as  the  amount 
of  enemy  casualties. 

This  method  gives  information  about  the  best  system  to  be  chosen  given  the 
assumptions  concerning  effectiveness  and  costs.  So  this  method  will  only  be  used 
when  systems  have  to  be  compared  on  cost  and  effectiveness  aspects. 

MCA  will  be  used  when  there  has  to  be  made  a  choice.  Questions  which  can  be 
answered  are  for  example  how  to  protect  the  own  troupes  to  get  the  least  victims  or 
how  to  collect  weapons  to  inn  the  most. 

These  models  can  be  used  for  almost  every  task,  because  almost  every  task  can  be 
optimised.  Only  the  tasks  which  are  exactly  described  by  standard  procedures  are 
excluded. 

These  models  can  be  used  for  almost  every  task,  because  almost  every  task  can  be 
optimised.  Only  the  tasks  which  are  exactly  described  by  standard  procedures  are 
excluded. 

Drama  theory  addresses  problems  involving  multiple  parlies  with  conflicting 
objectives.  This  is  a  variant  of  game  theory.  Drama  theory  also  takes  into  account  the 
way  people  play  games  at  different  levels  at  the  same  time.  So  for  every  task  where 
war  games  can  be  used,  drama  theory  can  be  used  too.  They  can  also  be  used 
where  information  is  needed  about  emotional  aspects. 

These  techniques  give  information  about  the  threat  of  the  opponent.  So  when 
information  is  needed  about  the  enemy  or  safety,  threat  assessment  can  be  used. 
These  networks  analyse  the  relationships  between  different  elements.  So  these  can 
give  information  about  relations  or  influences  between  people  or  actions. 


Table  3 

Schematic  overview  of  the  construction  of  the  matrix. 

ModeM  Model  2 

Model  i 

Task  1 

Task  2 

Task  i 
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The  qualifications  Logical  (L),  Theoretical  (T)  and  Practical  (P)  were  filled  in  the  cells 
for  each  combination  of  model*task.  The  complete  matrix  can  be  found  in 
Appendix  B. 

The  analysis  of  the  matrix  was  done  per  model  (vertical  analysis)  and  per  task 
(horizontal  analysis).  The  vertical  analysis  shows  applicability  of  the  models  in  general 
and  gives  an  indication  of  the  feasibility  of  a  specific  model  in  this  context.  The 
horizontal  analysis  shows  the  availabihty  of  models  to  solve  task-specific  research 
questions.  In  other  words,  it  shows  if  (and  how  many)  models  are  practically  available 
for  a  certain  task. 

4.1.1  Blanks 

40%  of  all  cells  in  the  matrix  were  left  blank.  These  blank  cells  in  the  matrix  represent 
combinations  of  tasks  and  models  that  could  not  be  categorized.  A  categorization  for 
these  methods  for  the  specific  task  does  not  seem  relevant  at  all.  For  example,  ‘war 
games’  do  not  seem  to  be  a  relevant  method  to  determine  the  result  or  (contribution  to) 
the  effect  of  an  information  task  (for  example  informing  civilians  on  mines).  The  total 
amount  of  (non)blank  cells  for  a  specific  model  gives  insight  in  the  applicability  of 
these  models  for  small  unit  operations  research. 

4.1.2  Logical 

For  33%  of  all  possible  combinations,  models  were  categorized  to  be  logical  (the 
highest  abstraction  level). 

4.1.3  Theoretical 

For  55%  of  all  combinations,  theoretical  feasible  methods  are  available.  The  expertise 
of  the  method  and  relevance  within  the  domain  are  recognized.  However,  the  domain 
specific  parameters  and  weights  are  not  available  (yet). 

4  A. 4  Practical 

The  inventory  of  models  shows  that  in  12%  of  all  possible  combinations  (models  X 
task)  models  that  are  tested  and  relevant  within  the  domain  are  available. 

4.2  The  Models 

When  performing  the  vertical  analysis,  we  can  roughly  consider  three  clusters  based  on 
the  amount  of  blanks,  L’s,  T’s,  P’s.  The  models  that  score  a  lot  of  blank  cells  or  L’s 
(logical),  seem  little  promising  in  solving  task  specific  research  questions  for  small  unit 
operation  research.  This  cluster  consists  of  ‘homogeneous  and  heterogeneous  combat 
models’,  ‘cost  effectiveness  analysis’,  ‘multiple  criteria  analysis’,  ‘drama  theory’  and 
‘quantitative  threat  assessment’  models  (modelhng  techniques)  ‘analytic  optimization’, 
‘heuristic  optimization’  ‘belief  networks  and  influence  diagrams’  are  models  (modelling 
techniques)  that  score  a  relatively  large  amount  of  T’s  (Theoretical),  which  means  that 
they  show  a  lot  of  potential  to  be  appUed  in  solving  task  specific  research  questions. 
However,  before  the  models  can  effectively  been  used,  a  considerable  amount  of 
empirical  testing  and  gathering  of  domain  specific  parameters  and  weights  has  to  be 
done. 

Based  on  the  total  amount  of  P’s,  (Practical)  the  ‘search  and  damage  assessment’, 
‘simulation  of  military  systems’  and  ‘war  game’  models  can  be  qualified  as  most 
promising.  For  these  models  there  is  knowledge  and  experience  available  at  TNO  and 
the  models  can  directly  be  applied  to  solve  research  questions. 

The  models  and  their  quahfications  are  summarized  in  Table  4. 
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Table  4  Models  and  their  qualifications. 


Little  promising 

;  Lot  of  potential 

Promising 

Homogeneous  and 

I  Analytic  optimization 

Search  and  damage 

heterogeneous  combat 
models 

assessment 

Cost  effectiveness  analysis 

Heuristic  Optimization 

Simulation  of  military  systems 

Multiple  Criteria  Analysis 

Drama  theory 

Quantitative  threat 
assessment;  static  and 
dynamic  analyses 

Belief  networks  and  Influence 
diagrams 

War  games 

4.3  Tasks 

When  considering  the  application  from  a  task  point  of  view,  the  qualification  of  the 
models  shows  the  amount  of  knowledge  and  expertise  for  specific  tasks. 

If  practical  applicable  models  per  task  are  available,  means  that  empirically  tested 
knowledge  is  available  for  this  task.  Logically,  it  is  desirable  to  have  at  least  one 
method  that  is  categorized  ‘practically’  for  each  task. 

When  the  amount  of  P’s  is  more  than  one  for  a  specific  task,  that  the  performance 
measurement  for  this  specific  task  has  yet  been  studied  from  more  than  one  perspective 
(e.g.  by  the  use  of  more  than  one  model).  The  methods  have  been  empirically  tested  in 
the  relevant  domain.  Hence,  the  use  of  more  than  one  relevant  model  increases  the 
vahdity  of  the  prediction  of  eventual  task  performance  (result). 

Analyzing  the  qualification  of  tasks  in  the  matrix,  shows  that  the  largest  amount  of  P’s 
can  was  found  for  combatant  tasks.  For  non-combatant  tasks,  there  is  hardly  any 
empirically  tested  data  available. 
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5  Discussion  and  conclusions  (synthesis) 


The  results  of  all  tasks  a  mission  comprises  of  have  to  be  evaluated  and  quantified  in 
order  to  make  Operational  Effectiveness  measurable  (Figure  2).  There  are  several 
models  available  to  quantify  the  results  of  a  task  as  discussed  in  the  chapters  before.  To 
evaluate  operational  effectiveness  of  a  mission,  at  least  one  model  should  be  able  to 
quantify  the  result  of  all  tasks  involved  in  that  mission. 

The  inventory  of  models,  in  combination  with  tasks,  performed  in  this  study  shows  that 
no  there  is  no  model  available  that  can  practically  or  be  used  to  quantify  all  tasks  for 
small  unit  operations.  The  most  promising  models  to  do  so  in  future  are  the  models  that 
only  score  P’s  or  T’s  in  the  inventory,  like  ‘simulation  of  military  systems’  and 
‘influence  diagrams  and  belief  networks’. 

Another  remarkable  result  of  the  inventory  is  that  the  expertise  and  knowledge  of  TNO 
is  available  for  combatant  tasks.  For  several  tasks,  like  ‘conduct  an  attack’  or  ‘perform  a 
tactical  movement’  data  and  numbers  are  available  and  4  different  types  of  models  can 
directly  be  applied  to  answer  research  questions  (4  P’s  and  5  T’s). 

Based  on  the  matrix  we  can  conclude  that,  in  general,  that  there  is  a  lack  of  knowledge 
and  expertise  on  non-combatant  tasks  at  TNO.  It  is  probably  also  true  that  this  gap  is  a 
world-wide  issue.  The  relations  between  variables  are  not  known  for  the  majority  of 
these  tasks.  In  OKE  inclusive  this  project  we  make  an  attempt  to  get  a  first  grasp  on 
defining  the  gap  and  start  closing  it. 
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A  Dutch-English  Translation  of  Tasks 


Taak 

Task 

Doel 

Goal 

Resultaat 

Result 

Inrichten 

waarnemingspost 

Construct 

observation  posts 

Verbindingen 

leggen; 

Slaapplaatsen 
creeeren;  Inrichten 

post 

Establish 

connections, 
create  sleeping 
accommodation, 

arrange  post 

Complete 

waarnemingspost 

ingericht 

Arrangement 
observation  post 
completed 

Bemannen 

waarnemingspost 

Man  observation 

posts 

Waarnemen  terrein 

en  opponent  en 
doorgeven  info 

Observe  terrain 

and  opponent; 

communicate 

intel 

Inlichlingen  over 

terrein  en 

opponent 

gewonnen 

Gathered  intel  on 

terrain  and 

opponent 

Verkennings- 

patrouille 

Reconnaissance 

patrol 

Waarnemen  terrein 

en  opponent  en 
doorgeven  info 

Obsen/e  terrain 

and  opponent; 

communicate 

intel 

Inlichtingen  over 

terrein  en 

opponent 

gewonnen 

Gathered  intel  on 

terrain  and 

opponent 

Gevechtspatrouille 

Combat  Patrol 

Uitschakelen 

dreiging 

Neutralize  threat 

Vijandelijke 

eenheden 

uitgeschakeld 

Hostile  forces  are 

neutralized 

Beveiligings- 

patrouille 

Security  Patrol 

Penetratie  vijand 
voorkomen; 
Beveiliging  eigen 

terreindelen 

Prevent  the 

enemy  to  access 

own  area;  secure 

own  terrain 

Vijandelijke 

penetratie  eigen 

gebied 

voorkomen; 

Veiligheid 

vergroot 

Access  of  hostile 

troops  prevented; 

increase  of 
security 

Contactpatrouille 

Contact-  Patrol 

Verbinding  tussen 

twee  eenheden 

Connection 

between  two  units 

Informatie 

uitgewisseld 

tussen  eenheden 

Information 

exchange 

between  units 

Mijn-  en  explosief 

opsporings- 

patrouille 

Patrol  to  locate 

mines  and 

explosives 

Mijnen/explosieven 

vinden 

Find  mines/- 

explosives 

In  kaart  gebracht 
mijnenveld; 

Gecontroleerde 

veilige  gebied 

Map  minefield: 

checked  secure 

area 

Sociale  patrouille 

Social  Patrol 

Contact  met 
bevolking 

Communication 

with  local 

population 

Inlichtingen  uit 
bevolking 

gewonnen; 

Vertrouwen 

gewonnen; 

Openheid 

bevolking  is 

vergroot; 

Toegankelijkheid 

bevolking  is 

vergroot 

Obtain 

intelligence  form 
local  population; 
gained  trust; 
openness  of 
population  is 

increased; 
accessibility  of 
population  is 

increased. 

Inspecteren  van 
voertuigen  en 

mensen 

(checkpoints) 

Inspect  vehicles 
and  persons/man 
checkpoints 

Op  de  hoogte 
blijven  verplaatsing 

Stay  informed  of 

movements 

Ongecontroleerde 

verplaatsing 

voorkomen 

Prevent 

uncontrolled 

movements 
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Taak 

Task 

Doel 

Goal 

Resultaat 

Result 

Inspecteren  van 
gebieden  en 
gebouwen  (site 
inspectie) 

Inspect  sites, 

areas  and 

buildings 

Controleren 

wapen  en 

munitievoorraden 

Monitor  weapon 

and  ammunition 

storage 

Niet  ingezette 

wapens 

Non-used 

weapons 

Afsluiten  en 

doorzoeken  van 

gebouwen  en 
gebieden 

Cordon  and 

search  sites, 

areas  and 

buildings 

Personen  en 

explosieven 
zoeken;  innemen 

van  gebouw 

Search  persons 
and  explosives: 
take  building 

Personen  en 

explosieven 

gevonden; 

gebouw/gebied 

ingenomen 

Found  persons 
and  explosives; 
take  building 

Beschermen  en 

beveiligen  van 
objecten, 
gebieden  en 

personen 

Monitor/protect 
objects,  areas 
and  persons 

Verzamelen, 

doorgeven 

gegevens  en 

penetratie  van 
vijand  voorkomen 

Collect  and  pass 

information  in 

order  to  prevent 

enemy  access 

Vijandelijke 
penetratie  eigen 
gebied 

voorkomen 

Prevented  hostile 

access  of  own 

area 

Escorteren, 

konvooi 

beveiliging/ 

bewaking 

Escort/convoy 
security  and 
protection 

Beveiligen 

verplaatsende 

eenheid 

Secure  moving 

units 

Veilig 

getransporteerde 

goederen/person 

en 

Secured 

transportation  of 
persons/goods 

Quick  reaction 

force 

Quick  reaction 

force 

Ondersteunen 

eenheid 

Support  unit 

Verhoogde 

operationele 

inzetbaarheid 

eenheid 

Increased 

operationality  of 

unit 

Wapeninzamel- 

acties 

Weapon 

collection 

activities 

Voorkomen  inzet 

wapens 

Prevent  (hostile) 
use  of  weapons 

Wapens 
ingezameld  en 
verhoogde 
veiligheid 

Weapons 

collected  and 

increased 

security 

Liaison 

uitbrengen 

(bemiddelen) 

Conduct  liaison 

Uitwisselen 

informatie 

Exchange 

information 

Verbeterde 

coordinatie 

tussen  alle 

parti]  en 

Improved 

coordination 

between  parties 

Showing  the 
force/flag 
(voorbeeld 
schietoefeningen) 

Showing  the 
force/flag  (e.g. 
shooting 
exercises) 

Sterkte  tonen 

Show  own 

capabilities/ 

strength 

Geintimideerde 

vijand  en 
verhoogd  gevoel 
van  veiligheid 

onder  de 

bevolking 

Enemy 

intimidated, 
sense  of  safety 
among  population 

is  increased 

Verspreiden  van 
Info  Ops 
producten 
(posters,  folders 
etc.) 

Spreading  of  Info 
Ops  products 
(e.g.  flyers) 

Voorlichten 

bevolking 

Inform  population 

Vergroten 
veiligheidsgevoel 
bij  bevolking 

Increase  sense  of 
safety  among 
population 

Voorlichten  van 

burgers  en  lokale 
organisaties  in 

mine  awareness 

Inform  cilvilians 

and  local 

organisations 

about  mine 

awareness 

Voorlichten 

bevolking  over 
mijnen 

Inform  population 
on  mines  (mine- 
awareness) 

Voorkomen 

mijnongevallen, 
goodwill  kweken 
bij  bevolking 

Prevent  mine- 

accidents,  create 
goodwill  among 
population 
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Taak 

Task 

Doel 

Goal 

Resultaat 

Result 

Verkenning  met 

andere  middelen 

dan  patrouilles 
(RPV,  EOV,  radar 
etc.) 

Reconnaissance 

with  other  means 

than  patrols 
(RPV,  EOV, 

Radar,  etc.) 

Waarnemen 

terrain,  info 
doorgeven 

Observe  terrain, 

pass/exchange 

information 

Inlichtingen  over 

terrein  en 

opponent 

gewonnen 

Gathered 

information  on 

terrain  and 

opponent 

Ondersteuning 
leveren  bij  het 

herstellen  van  de 

lokale 

infrastructuur 

Provide 

assistance  in 

rebuilding  the 

local 

infrastructure 

Infrastructuur 

verbeteren 

Improve 

infrastructure 

Infrastructuur 

verbeterd;  Woon 

en  leef  klimaat 

verbeterd 

Improved 

Infrastructuur: 

Living  conditions 
are  improved 

Aanvoer  van 

klasse  1.  II.  Ill,  IV. 

V 

Supply  class  1,  II, 

III,  IV,  V 

Goederen 

transporteren 

Transport  goods 

Voorraden 

bijgevuld 

Replenished 

stocks 

Instellen  van 
verbindingen 

(Re-)establishing 

communications 

Communicatie 

tussen  eenheden 

Communication 

between  units 

Eenvoudige 

informatie 

uitwisseling 

Information 

exchange 

Leveren  van 

transportsteun 

(voertuigen, 

chauffeurs) 

Transport  support 
(drivers,  vehicles) 

Transportcapacteit 

vergroten 

Increase  the 

capacity  of 
transport 

Meer  goederen/ 

eenheden 

getransporteerd 

Increase  in 

transportation  of 
goods/units 

Ondersteuning 

leveren  voor 

Crowd  and  Riot 

Control  (CRC) 
operatie 

Support  to  Crowd 
and  Riot  Control 

(CRC)  operations 

Escalatie 

bevolkingsopstand 

voorkomen 

Prevent  riotering 
of  population 

Gecontroleerde 

bevolking 

Population  under 

control 

Ondersteuning 
leveren  bij 

arrestatie 

(bewaking  en 
bescherming) 

Support  to  arrests 
(security  and 
protection) 

Escalatie  tijdens 

arrestatie 

voorkomen 

Inhibit  escalation 

of  aggression 
during  arrest 

Gecontroleerde 

arrestatie 

Controlled  arrest 

Cobrdinatie  met 

aangrenzende 

eenheden 

Transport  support 
(drivers,  vehicles) 

Verbinding  tussen 

twee  eenheden 

onderhouden 

Maintain 

connection 

between  units 

Informatie 

uitgewisseld 

verbeterde 

taakuitvoerinq 

Information 

exchange; 
improved  task 
performance 

Innemen  van  een 

opstelling 

Take  a  position 

Verdedigen 

terreindeel  door 

middel  van 

uitbrengen  vuur 

Defend  terrain  by 
firing 

Gebied 

behouden, 

overgenomen 

initiatief 

Kept  own  area, 

took  over  initiative 

Uitvoeren  van 

een  vuuroverval 

Conduct  a  fire 

raid 

Uitschakelen 

vijandelijke 

eenheden 

Neutralize  hostile 

units 

Vijandelijke 

eenheden 

uitqeschakeld 

Hostile  units 

neutralized 

Uitvoeren  van 

een  aanval 

(bereden) 

Conduct  a 

mounted  attack 

Gebied  veroveren; 
zuiveren  gebied 
(uitschakelen 
vijandelijke 
eenheden) 

Claim  area;  clear 
area  (neutralize 
hostile  units) 

Gebied  veroverd, 
vijandelijke 

eenheden 

uitgeschakeld) 

Area  taken, 

hostile  units 

neutralized 
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Taak 

Task 

Doel 

Goal 

Resultaat 

Result 

Uitvoeren  van 

een  tegenaanval 

Conduct  a 

counter  attack 

Hernemen  gebied; 
uitschakelen 

vijandelijke 

eenheden 

Reclaim  area, 

neutralize  hostile 

units 

Gebied  heroverd; 
vijandelijke 

eenheden 

uitgeschakeld 

Area  taken, 

hostile  units 

neutralized 

Uitvoeren  van 

een  tegenstoot 

Conduct  a 

counter  strike 

Uitschakelen 

vijandelijke 
eenheden,  Vijand 
tijdelijk  initiatief 

ontnemen 

Neutralize  hostile 

units,  take  over 

initiative 

Vijandelijke 

eenheden 

uitgeschakeld, 

Tijd  en  initiatief 

gewonnen 

Hostiles 

neutralized, 
regain  initiative 

Uitvoeren  van 

een  versterking 

Increase  fighting 
strength 

Gevechtskracht 

vergroten 

Increase  fighting 
strength 

Gevechtskracht 

vergroten 

Fighting  strength 

increased 

Uitvoeren  van 

een  vuurbasis 

Establish  a  fire 

base 

Ondersteunen 

manouvre  eenheid, 
suppressie  van 
vijandelijke 

eenheden 

Support  units 

maneuvers, 

suppress  hostile 

units 

Gesuppresseerde 

vijandelijke 

eenheden 

Hostile  units  kept 

under 

suppression 

Innemen  van  een 

teamverzamel* 

gebied 

establish  a  team 

collection  area 

Verzamelen 

eenheden 

Assemble  units 

Gehergroepeerde 

eenheden 

Assembled  units 

Uitvoeren  van 

een  tactische 

verplaatsing 

Perform  a  tactical 

movement 

Verplaatsen  door 
terrein, 

uitschakelen 

vijandelijke 

eenheden 

Move  through 
terrain,  neutralize 

hostile  units 

Eenheid 

verplaatst, 

vijandelijke 

eenheden 

uitgeschakeld 

Unit  moved, 

hostile  units 

neutralized 

Uitvoeren  van 

een  infiitratie  of 

exfiltratie 

Perform  an 

infiltration  or 

exfiltration 

Onopgemerkte 
verplaatsing  door 
vijandelijk  gebied; 
contact  met  eigen 

eenheden  maken 

Unnoticed 

movement 

through  hostile 
area,  contact  with 

own  units 

Eenheid 

verplaatst; 
gegevens  terrein 
en  vijand;  gebied 
bezet;  vijand 
misleid;  eenheid 
teruggekeerd  in 
eigen  gebied 

Unit  moved. 

information 

terrain  and 

enemy,  area 

taken,  misled 
enemy,  unit 

returned  to  own 

area 

Uitvoeren  van 

een  aanval  te 

voet  of  een 

uitgestegen 

aanval 

Conduct  an 

attack  on  foot  or  a 

dismounted 

attack 

Gebied  veroveren; 
zuiveren  gebied 
(uitschakelen 
vijandelijke 
eenheden) 

Conquer  area; 

clear  area 

(neutralize  hostile 
units) 

Gebied  veroverd, 
vijandelijke 

eenheden 

uitgeschakeld 

Area  taken, 

hostile  units 

neutralized 

Uitvoeren  van 

een  opmars  te 

voet 

Conduct  an 

advance 

Verplaatsen  door 

terrein 

Move  through 

terrain 

Eenheid 

verplaatst 

Unit  moved 

Data  matrix  tasks* methods 
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C  Research  methods,  models  &  simulations 


Research  methods 


Experiments 


Description 

Variables  are  qualities  a  researcher  wants  to  study  and  draw  conclusions  about.  If  one 
wants  to  determine  how  two  variables  relate  to  each  other  or  how  one  variable 
influences  another  (causal  relationship),  the  most  appropriate  research  method  is  often 
an  experiment.  Experiments  are  not  only  extremely  suitable  to  determining  causal 
relationships,  but  can  also  be  used  for  the  comparison  of  two  (or  more)  different 
treatments,  concepts  or  strategies.  Experiments  can  be  divided  into  two  types:  random 
or  true  experiments,  and  quasi  experiments. 

Performing  an  experiment  can  be  described  as  ‘a  controlled  method  of  observation  in 
which  the  value  of  one  or  more  independent  variables  is  manipulated  in  order  to 
determine  causal  effects  on  one  or  more  dependent  variables’  (Zechmeister  et  al.,  200 1 ). 
Experimentation  does  not  so  much  refer  to  a  specific  location  or  room  in  which  a  study 
is  performed,  but  rather  to  the  rules  and  assumptions  that  apply  to  the  set-up  of  a  true 
experimental  design.  Within  experimental  research,  the  researcher  creates  an  artificial 
situation  in  order  to  ascertain  that  the  effects  found  are  due  to  the  manipulations  and  not 
to  other  (confounding)  factors.  The  researcher  controls  who  participates  in  the 
experiment,  what  happens  during  the  experiment  and  the  circumstances  under  which 
the  experiment  takes  place. 

Variables 

Researchers  identify  the  variables  they  will  investigate  in  their  research.  In  experiments, 
two  types  of  variables  are  defined:  the  input  variables  that  are  manipulated  or  controlled 
(independent  variables)  and  the  output  variables  that  are  measured  (the  dependent 
variables).  Independent  variables  are  the  causes  and  dependent  variables  are  the  effects 
(or  measures  of  effects).  Independent  variables  must  have  at  least  two  levels  that  can  be 
compared  or  a  treatment  condition  versus  a  control  condition.  Dependent  variables  must 
be  measurable.  Beyond  discovering  causal  relationships,  experimental  research  further 
seeks  out  how  much  cause  will  produce  how  much  effect. 

Experimental  designs 

Several  designs  are  available  and  well  described  in  methods  books  that  can  be  applied 
to  true  experimental  designs.  Fairly  well  known  designs  are  for  example  a  ‘repeated 
measures  design’  and  a  ‘factorial  design’. 

A  repeated  measure  (or  within-subjects  design)  means  that  the  same  variable  is 
measured  with  the  same  subjects,  under  different  conditions  or  at  different  points  in 
time.  Each  time  individuals  participate  in  the  conditions  in  the  experiment,  they 
complete  the  dependent  variable  measure,  and  thus  the  measure  is  repeated. 

Experiments  in  which  two  or  more  independent  variables  are  studied  simultaneously, 
researchers  most  often  use  complex  designs  like  a  factorial  design.  In  factorial  designs, 
the  levels  for  each  independent  variable  are  combined  factorial.  Factorial  combination 
involves  creating  conditions  of  the  experiment  by  pairing  each  level  of  the  independent 
variable  with  each  level  of  a  subsequent  independent  variable. 
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Assumptions 

Randomized  experiments 

Some  strict  rules  apply  before  a  randomized  experiment  can  be  executed: 

Randomized  assignment  of  subjects  to  the  test  groups  or  of  test  conditions  per  subject 
(each  person  has  an  equal  chance  of  being  assigned  to  any  particular  condition). 

The  independent  variables  have  to  be  operationalized  and  have  more  than  one  level 
(e.g.  naen  and  women). 

It  must  be  possible  to  hold  other  variables  constant  or  to  assume  a  random  distribution 
of  the  variable  across  the  study  sample. 

Quasi  experiments 

Generally  speaking,  quasi-experiments  include  an  intervention  or  treatment  and  provide 
a  comparison,  but  they  lack  the  degree  of  control  found  in  true  experiments. 

Experiments  are  ‘quasi  experimental’  when  randomly  assigned  treatment  and/or 
comparison  groups,  or  ‘laboratory  control’  is  not  available.  Quasi-experimental  designs 
are  frequently  used  in  applied  research.  One  of  the  advantages  of  quasi  experiments  is 
that  a  researcher  can  make  use  of  existing  interventions  in  a  natural  situation.  The 
downside  is  that  it  is  usually  difficult  to  control  for  (interfering)  external  factors  because 
random  assignment  is  not  possible. 

Output 

The  data  generated  by  an  experiment  is  quantitative  by  definition  because  the  dependent 
variables  must  be  quantitatively  measurable.  Usually,  analyses  of  variance  are  the 
primary  statistical  technique  with  which  to  analyze  the  data. 

Application 

Experiments  are  mainly  performed  to  evaluate  cause  and  effect  relationships,  for 
example,  the  effect  of  a  certain  treatment  on  a  person’s  performance.  Experimental 
designs  are  also  frequently  used  to  compare  different  situations  or  new  or  improved 
concepts.  In  this  case,  different  types  of  concepts  are  compared  with  one  another. 
Experiments  can  be  of  great  use  for  the  Dutch  Defence  Organization  in  order  to  gather 
quantitative  data.  Comparison  studies  in  which  the  effects  new  (concepts  of)  equipment 
and  apparel  on  operational  performance  are  studied  helps  the  decision  making  process 
of  material  procurement  and  the  employment  of  personnel.  Experiments  can  also  serve 
in  the  analysis  of  the  effects  of  tactical  changes  in  physical,  physiological,  cognitive  and 
tactical  parameters  in  specific  operational  contexts,  on  operational  effectiveness. 

Advantages 

An  experiment  is  (one  of)  the  best  tools  with  which  to  analyze  causal  relationships. 

The  internal  validity  of  an  experiment  is  high  compared  to  other  data  research  methods. 
If  performed  well,  an  experiment  is  replicable  since  one  can  account  for  the 
experimental  conditions. 

Disadvantages 

The  data  that  one  gathers  within  an  experiment  applies  to  the  specific  situation  in  which 
that  experiment  was  performed.  Therefore,  it  can  be  difficult  to  generalize  the  results  to 
other  situations:  in  other  words,  the  external  validity  is  often  low.  However,  it  practice 
the  seriousness  of  this  disadvantage  depends  on  the  type  of  research  question. 
Performing  an  experiment  is  relatively  expensive  compared  to  other  data  collection 
methods  (e.g.,  questionnaires). 
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Track  record  TNO 

TNO  has  a  broad  expertise  in  the  area  of  experimental  research.  TNO  has  substantial 
resources  (knowledge  and  facilities)  to  perform  a  broad  range  of  experiments. 

Human  vulnerability  studies. 

Ballistic  impact. 

NBC  protection,  both  physical  as  physiological. 

The  effect  of  nutrition  on  performance. 

Experiments  performed  in  an  operational  context  (aiming  sights). 

Decision-making  processes. 

Optimal  communication  media. 

Field  studies 

Description 

A  field  study  is  a  research  method  that  can  be  used  for  the  description,  interpretation 
and  explanation  of  behaviours,  opinions  and  ‘products’  of  the  persons  concerned  in  an 
existing  situation  (the  field)  via  direct  data  collection  by  the  researcher.  The  researcher 
does  not  intervene  in  the  situation.  The  events  and  behaviours  occur  naturally  and  are 
not  manipulated  or  controlled  by  the  observer. 

Since  this  type  of  research  method  is  frequently  used  when  there  is  little  specific 
information  available  about  a  certain  research  problem,  the  goal  of  a  field  study  usually 
has  a  descriptive  nature. 

In  choosing  what  case  to  study,  a  researcher  may  choose  a  case  because  it  shows 
different  perspectives  on  the  problem,  process,  or  event  of  interest,  or  it  may  be  just  an 
ordinary  case,  accessible,  or  unusual. 

Some  researchers  performing  field  studies  participate  fully  in  the  setting  or  in  the  lives 
of  the  people  they  study,  by  either  being  or  becoming  members  of  the  group.  Others 
remain  outsiders,  purely  observers. 

The  data  collection  is  extensive,  drawing  on  multiple  sources  of  data  collection  methods 
(described  in  later  chapters)  such  as  observations,  interviews,  documents,  and  audio¬ 
visual  materials. 

Assumptions 

Variables  are  not  systematically  manipulated. 

An  object  of  research  should  always  be  studied  in  its  natural  environment, 
(‘contextuality’). 

It  is  not  permitted  to  reduce  the  research  situation  to  a  few  specific 
characteristics/dimension.  The  topic  of  study  should  be  covered  as  broadly  as  possible 
in  order  to  take  into  account  as  many  confounding  variables  as  possible  (holism). 

Output  data 

Field  studies  often  provide  qualitative  data  of  specific  situations.  However,  some  data 
collection  methods  used  within  a  field  study  can  produce  quantitative  data  (for  example 
a  survey). 

Application 

The  defence  organization  can  have  benefit  from  this  research  method  because  it  is 
extremely  suitable  for  augmenting  knowledge  of  operational  functioning  of  employees. 
Since  it  is  a  relatively  unobtrusive  research  method,  a  study  can  be  conducted  during 
field  exercises  or  operations  in  order  to  gain  insight  into  military  procedures  or  tactics. 

If,  for  example,  new  strategies  or  equipment  are  being  tested  by  the  military  during 
training,  the  combination  of  objective  observations  by  an  ‘outsider’  (researcher)  and,  for 
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example,  a  survey  produces  a  large  amount  of  both  qualitative  and  quantitative 
information. 

Advantages 

One  can  collect  detailed  information  about  very  specific,  real  life  situations. 

Field  studies  can  be  a  rich  source  of  information. 

It  is  possible  to  study  a  phenomenon  in  great  detail. 

Can  focus  on  rare,  unique,  and  extreme  cases. 

Disadvantages 
Time  consuming. 

It  may  violate  people's  privacy. 

Data  collection  is  vulnerable  to  biases. 

Track  record  TNO 

CRC  study  of  psychological  processes  during  training  exercises. 

Study  on  types  of  psychological  operations. 

Exploratory  study  on  the  use  of  information  operations  in  the  field. 

Archival  research 

Description 

For  some  types  of  research  questions  it  is  simply  impossible  to  conduct  an  experiment 
or  field  study.  It  is,  for  example,  impossible  to  infer  the  relationship  between 
environmental  conditions  (temperature)  and  mortality  rate  from  experimental  or  field 
studies.  Disasters  are  also  phenomena  that  can’t  be  studied  by  conducting  an  experiment 
or  performing  a  field  study.  Specific  research  questions  on  issues  like  these,  can  only  be 
answered  by  the  analysis  of  existing  data. 

New  data  is  not  being  collected  in  archival  research,  unlike  experimental  research  or 
field  studies,  archival  research  includes  the  analysis  or  integration  of  existing  data. 
Existing  data  can  consist  of  (the  sediment)  of  verbal  or  non-verbal  behaviour,  in 
writing,  image  or  sound.  The  nature  of  the  archive  that  acts  as  data  sources  can  roughly 
be  divided  in  four  categories  (the  data  collection  methods  concerned  with  these 
categories  will  be  explained  in  Chapter  3.3). 

Statistical  records  and  other  government  or  institutional  records,  such  as  daily 
temperatures  or  population  growth. 

Survey  archives  of  data  collected  for  research  purposes  by  standard  survey  methods, 
such  as  socio  demographic  information  or  attitudes  on  social  and  political  topics. 

Written  records,  or  other  media  such  as  private  diaries,  newspapers,  sound  archives  or 
videotapes. 

One  of  the  data  collection  methods  that  can  be  used  within  archival  research  is  the 
Meta-analysis. 

Assumptions 

Analyses  of  existing  data  may  be  performed  if: 

•  there  are  sufficient  and  relevant  data  available  to  analyze; 

•  the  data  form  valid  indicators  of  the  concepts  being  studied. 

Output  data 

The  output  from  the  analysis  of  ex.isting  data  is  dependent  upon  the  type  of  data 
analyzed  and  the  method  used  to  evaluate  or  integrate  the  data.  Some  methods  yield 
quantitative  data,  such  as  the  use  of  statistical  records  or  meta-analyses.  Others  are  more 
descriptive  in  their  product,  such  as  a  case  study  or  the  evaluation  of  written  records. 
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Even  these  qualitative  methods  may  yield  quantitative  data  if  the  data  is  evaluated  using 
quantitative  measures  (a  scorecard  method,  for  example). 

Application 

The  Dutch  Defence  Organization  can  benefit  from  archival  research  in  several  ways, 
like  analysis  of  strategies  used  during  former  operations  or  retrieval  of  knowledge  about 
the  most  recent  developments  on  for  example  training  methods  by  performing  a 
literature  study  or  meta-analysis. 

Advantages 

High  external  validity. 

Experimenter  cannot  affect  behaviour. 

Large  volume  of  data  enables  relationships  to  emerge. 

Archival  research  is  sometimes  labelled  as  relatively  cheap  because  the  researcher  does 
Not  have  to  invest  the  time  and  cost  involved  in  data  collection  and  recording. 

However,  retrieving  the  appropriate  data  can  cost  large  amounts  of  time  too. 

Disadvantages 

Researcher  has  no  control  over  how  data  are  collected. 

Important  information  may  not  have  been  recorded  and  can  easily  affect  the  internal 
validity. 

Archival  research  is  not  a  suitable  research  method  to  determine  causal  relationships. 
Correlations  between  variables  can  be  determined. 

Experimenter  cannot  manipulate  behaviour. 

Track  record  TNO 

Crowd  &  Riot  control  for  the  Dutch  Military  Pohce  (KMAR). 

Research  on  social  security. 

Automotive  behaviour  (parking). 

Historical  research  e.g.  Srebrenica  or  other  conflicts  (Belinda  Smeenk). 

Data  collection  methods  Interviews 

Description 

Interviews  provide  in-depth  information  about  a  particular  research  issue  or 
question.  Because  the  information  is  not  quantifiable  (i.e.,  not  amenable  to 
statistical  analysis),  the  interview  often  is  described  as  a  qualitative  research 
method.  Whereas  quantitative  research  methods  (e.g.,  experiments)  gather  a  small 
amount  of  information  from  many  subjects,  interviews  often  gather  a  broad  range  of 
information  from  a  few  subjects.  An  interview  is  an  interactive  process,  in  which 
the  interviewer  and  respondent  can  influence  each  other.  An  interviewer  has  the 
dual  goals  of  motivating  the  respondent  to  give  full  and  precise  replies,  which  may 
take  effort  (e.g.,  probing  the  respondent’s  memory  for  specific  details  or  dates),  and 
avoiding  biases  stemming  from  social  desirability,  conformity,  or  other  sources.  In 
general  it  is  the  interviewer  who  must  attempt  to  fulfil  these  goals  in  interaction 
which  each  individual  respondent.  The  specific  tasks  of  the  interviewer,  which 
contribute  to  these  goals,  involve  creating  a  positive  atmosphere,  asking  the 
questions  properly,  obtaining  an  adequate  response,  recording  the  response,  and 
avoiding  biases. 

Structured  versus  unstructured  interviews 

In  a  structured  interview,  the  interviewer  asks  a  specific  set  of  questions.  In  the 
structured  standardized  interview,  these  questions  are  printed  and  the  interviewer 
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asks  or  reads  the  questions  in  a  specific  order  or  sequence.  Each  question  has  been 
carefully  pre-tested  to  express  the  precise  meaning  desired  in  as  simple  a  manner  as 
possible.  Ln  the  unstructured  interview,  there  are  no  specific  questions  or  guidelines 
for  the  interviewer  to  follow.  Thus  each  unstructured  interview  is  unique. 
Unstructured  interviews  provide  considerable  flexibility  but  at  the  expense  of 
stability  of  the  interview  data. 

Types  of  interviews 

In  the  focused  interview,  the  main  function  of  the  interviewer  is  to  focus  attention 
on  a  given  experience  and  its  effects.  Interviewers  know  in  advance  what  topics  or 
what  aspects  of  a  question  they  wish  to  cover.  Somewhat  similar  to  the  focused 
interview  is  the  clinical  interview,  the  primary  difference  is  being  that  the  latter  is 
concerned  with  broad  underlying  feelings  or  motivations  or  with  the  course  of  the 
individual’s  life  experiences  rather  with  the  effects  of  a  specific  experience.  In  the 
non-directive  interview,  the  initiative  is  more  in  the  hands  of  the  respondent. 
Although  interviewers  are  expected  to  ask  questions  about  a  given  topic,  they  are 
instructed  not  to  bias  or  direct  the  respondent  to  one  rather  than  to  another  response. 
In  this  form  of  interviewing  the  function  of  the  interviewer  is  simply  to  encourage 
the  respondent  to  talk  about  a  given  topic  with  a  minimum  of  direct  questioning  or 
guidance. 

Assumptions 

Research  using  interviews  assumes  that: 

•  respondents  are  able  to  answer  the  questions  posed  to  them; 

•  respondents  are  willing  to  cooperate,  and 

•  the  research  question  of  interest  can  be  answered  using  the  type  of  data  resulting 
from  interviews. 

Output 

The  data  resulting  from  interviews  is  generally  qualitative,  that  is  to  say  descriptive. 
This  is  particularly  the  case  with  unstructured  interviews.  In  order  to  analyze  this  type 
of  data,  methods  are  used  to  categorize  the  information  so  it  can  be  quantified. 

When  interviews  are  structured  and  when  close-ended  questions  are  used,  the  resulting 
data  is  quantitative.  Many  methods  are  available  with  which  the  data  can  be  analyzed. 
The  choice  of  method  depends  upon  the  way  interviewees  are  able  to  respond  to  the 
questions  (for  example,  in  terms  of  measurement  scale). 

Application 

Interviews  are  particularly  useful  in  exploratory  situations.  That  is,  when  research 
questions  cannot  be  stated  in  any  great  detail  because  not  enough  is  known  about  the 
phenomena  being  studied. 

When  the  topic  of  study  is  ambiguous  or  difficult  to  succinctly  clarify  in  a  written 
questionnaire,  interviews  are  often  helpful. 

When  in  depth  information  is  desired  regarding  the  topic  of  study,  which  cannot  be 
achieved  with  a  written  questionnaire,  interviews  are  preferable. 

Advantages 

The  interviewer  has  the  ability  to  notice  and  correct  misunderstandings,  to  probe 
inadequate  or  vague  responses,  and  to  answer  questions  and  allay  concerns,  which  are 
important  in  obtaining  complete  and  meaningful  data. 

The  interviewer  can  control  the  context  of  the  interview,  including  the  possible  biasing 
presence  of  other  people. 
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Personal  interviews  can  attain  the  highest  response  rate  of  any  survey  technique. 
Moreover,  a  face-to-face  interviewer  can  establish  a  good  rapport  and  motivate  the 
respondent  to  answer  fully  and  accurately,  again  improving  the  quality  of  the  data. 

In  unstructured  interviews  the  interviewer  can  control  the  order  in  which  the  respondent 
receives  the  questions,  something  that  is  not  possible  in  written  questionnaires. 

Disadvantages 

As  a  result  of  the  intense  contact  between  interviewer  and  respondent,  interviewer 
effects  may  occur  that  can  be  detrimental  to  the  data.  The  interviewer’s  expectations  or 
personal  characteristics  (such  as  gender  or  race)  can  influence  responses. 

Personal  interviews  are  costly  because  interviewers  need  to  be  hired  and  trained;  travel 
expenses  can  be  substantial;  and  interviews  might  be  restricted  to  a  fairly  small 
geographic  area  because  of  the  logistical  problems  of  sending  interviewers  long 
distances. 

When  the  content  of  interest  might  prove  embarrassing  or  threatening  to  the  respondent 
or  is  viewed  as  private  or  personal,  direct  questioning  may  elicit  deceptive  responses  or 
refusals  to  answer.  The  basis  on  which  achieving  valid  responses  to  all  sensitive 
questions  (and  non-sensitive  questions  as  well)  depends,  is  the  interviewer’s  rapport 
with  the  respondent.  Skilled  interviewers  encounter  remarkably  few  refusals  to  reply. 
There  are  two  primary  sources  of  error  in  the  interview:  those  pertaining  to  the  validity 
or  meaning  of  interview  data  and  those  pertaining  to  the  dependability  or  reliability  of 
interview  data.  The  tendency  to  draw  general  conclusions  about  an  individual  based  on 
the  limited  data  of  a  first  impression  limits  tlie  meaning  and  accuracy  of  interview  data. 
Furthermore,  the  predictive  validity  for  interview  data  varies  widely  (i.e.,  the  ability  to 
generalize  results  to  similar  situations).  The  reliability  of  interview  data  has  been 
measured  primarily  in  terms  of  agreement  among  interviewers  on  relevant  variables. 

The  more  structured  the  interview,  the  more  interviewers  agree. 

Track  record  TNO 

TNO  has  a  substantial  amount  of  experience  with  interview  research.  A  few  examples 
are: 

•  performance  and  satisfaction  in  teams; 

•  experience  with  information  operations; 

•  the  effects  of  psychological  operations. 

Data  collection  methods  Observations 

Description 

Observation  is  a  method  by  which  information  on  behaviour  can  be  collected  through 
watching  and  recording  what  subjects  do  in  situations  of  interest.  Observation  makes 
use  of  a  more  naturalistic  setting  than  is  available  with  other  types  of  methods,  such  as 
surveys  or  interviews  (i.e.,  the  behaviour  is  performed  in  a  natural  way  in  natural 
surroundings).  This  is  the  result  of  the  relatively  unobtrusive  nature  of  observations  on 
the  one  hand  (observations  usually  do  not  disturb  the  subjects  while  they  are  performing 
their  behaviour),  and  the  potential  for  insight  into  behaviour  which  may  be  difficult  for 
a  participant  to  describe  on  the  other.  Observations  occur  along  a  continuum  ranging 
from  complete  immersion  in  the  group  being  observed  (infiltration)  so  that  the  subjects 
of  the  observation  are  not  aware  they  are  being  observed,  to  complete  transparency  in 
which  the  subjects  know  they  are  in  a  setting  in  which  their  behaviour  is  being  observed 
and  who  is  doing  the  observing.  Somewhere  in  the  middle  of  the  continuum  are,  for 
example,  observation  strategies  such  as  may  be  used  when  observing  the  number  of 
people  who  visit  a  public  place.  In  this  case,  no  attempt  is  made  to  hide  the  fact  that 
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subjects  are  being  observed  (the  observer  may  be  seated  in  clear  view,  obviously 
making  notes),  but  it  is  not  announced  to  those  being  observed  either. 

Informal  observations  can  be  made  in  almost  all  situations,  but  observations  are 
scientific  when. 

The  observations  serve  a  formulated  research  purpose. 

The  observations  are  deliberately  planned. 

The  observations  are  systematically  recorded. 

The  observations  are  subjected  to  checks  and  controls  regarding  validity  and  reliability. 
Generally  speaking,  we  can  distinguish  between  two  types  of  observations:  qualitative 
observation  and  quantitative  observation. 

In  qualitative  observation,  the  primary  goal  of  the  study  is  usually  not  to  collect  data  as 
such,  but  to  generate  hypotheses  in  an  exploratory  manner,  which  may  function  as  input 
for  other,  more  structured  future  studies.  This  may  be  done  using  participant 
observation,  in  which  the  observer  'infiltrates’  the  group  that  is  being  observed  and 
does  not  make  formally  known  that  he/she  is  an  observer  or  that  the  subjects  are  being 
observed. 

In  quantitative  observation,  the  goal  is  to  test  a  hypothesis  using  the  systematic  tallying 
of  behaviours  or  other  information  of  interest.  In  quantitative  observation,  behaviours 
may  be  directly  observed  and  tallied  -  or  counted  -  according  to  a  pre-determined 
category  system,  such  as  the  number  of  times  a  soldier  in  the  field  consults  a  compass 
or  other  localization  device.  Behaviour  may  also  be  assessed  by  examining  the 
frequency  of  a  particular  piece  of  information  not  specifically  produced  within  the 
limits  of  the  study  (in  terms  of  either  time  or  space). 

Observations  may  be  registered  via: 

Field  notes  in  which  the  observer  notes  everything  he  or  she  observes,  in  an  open, 
descriptive  format  (i.e.  not  constrained  by  categories  or  other  registration  structures). 
Field  notes  may  be,  for  example,  written  notes  or  verbal  recordings. 

Checklists  or  similar  tools  which  allow  the  occurrence  of  specific  behaviours  to  be 
tallied  and  quantified. 

Assumptions 

Observations  can  be  used  as  a  method  of  data  collection  if. 

Observable  measures  indicative  of  the  construct  of  interest  can  be  defined. 

A  suitable  setting  for  observation  can  be  found  in  which  the  behaviour  occurs 
frequently  enough  to  make  registration  worthwhile,  and 

Observers  can  be  trained  to  accurately  identify  the  relevant  behaviours  when  they  occur. 
Output 

Depending  upon  the  type  of  observations  conducted,  different  types  of  data  output  are 
possible.  When  observations  are  qualitative,  qualitative  data  is  the  result,  which  is 
generally  descriptive  in  nature.  Analysis  techniques  are  available  through  with  this  type 
of  data  can  be  categorized,  however,  the  reliability  of  such  a  resulting  quantification  is 
dependent  upon,  for  instance,  the  number  of  observations. 

When  observations  are  quantitative,  the  resulting  data  is  numerically  quantified,  though 
the  applicable  measurement  scale  (nominal,  ordinal,  interval,  and  ratio)  may  differ, 
depending  on  the  protocol  used  to  score  the  observed  behaviours. 

Application 

Observations  are  used  to  study  overt  behaviour.  When  the  goal  of  a  study  is  to 
understand  what  people  (or  animals,  for  that  matter)  do  in  a  particular  environment  or  in 
response  to  a  particular  stimulus,  and  the  use  of  a  self-report  measure  is  not  ideal 
(monkeys  cannot  give  interviews),  observations  are  a  preferable  method. 
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Observations  can  be  used  to  collect  data  in  situations  in  which  the  concept  of  interest 
can  be  operationalized  in  terms  of  overt  behaviours  that  can  be  perceived  by  an 
observer.  For  example,  it  is  possible  to  indirectly  assess  the  influence  of  a  peace¬ 
keeping  force  on  the  local  population  by  counting  the  number  of  weapons  turned  in 
during  a  weapons-for-cash  campaign. 

Advantages 

It  is  possible  to  study  and  describe  naturally  occurring  events  in  their  natural  settings. 
Respondents  are  relatively  unaware  that  they  are  being  studied. 

Activities  can  be  studied  that  are  difficult  for  respondents  to  describe  (such  as  nonverbal 
behaviour)  or  that  respondents  feel  uncomfortable  talking  about  (compliance  with  rules 
and  regulations). 

Observations  are  well  suited  to  research  in  which  measuring  the  behaviour  of  interest 
would  otherwise  affect  the  performance  of  the  behaviour. 

Observations  can  be  used  in  situations  in  which  data  collection  via  other  methods  is 
relatively  difficult  (e.g.  the  group  of  interest  cannot  be  expected  to  fill  in  questionnaires 
or  give  interviews). 

Observations  can  be  made  in  the  field,  but  are  also  suitable  for  use  in  a  laboratory 
setting. 

Observational  studies,  depending  on  the  way  the  study  is  designed,  can  be  carried  out 
for  a  relatively  low  budget. 

Disadvantages 

Due  to  the  subjective  nature  of  much  observational  research,  the  reliability^  of  the 
observations  may  be  lower  than  desired. 

An  observed  behaviour  is  intended  to  be  indicative  of  another,  more  abstract  construct. 
If  the  behaviour  is  indeed  a  good  indicator  of  that  construct,  the  study  has  good  validity. 
However,  observations  measure  constructs  in  an  indirect  manner  by  their  very  nature. 
Consequently,  there  is  a  considerable  risk  that  the  observed  behaviour  is  actually  an 
indicator  of  a  dimension  other  than  the  construct  of  interest. 

Observations  made  without  the  consent  of  those  observed  may  pose  ethical  problems. 
Factors  such  as  the  identifiably  of  those  observed,  whether  the  subject  would  actually 
want  to  be  observed  or  not  and  the  consequences  of  the  results  of  the  study  for  relevant 
individuals  must  be  considered. 

In  situations  in  which  the  observer’s  presence  is  either  too  obvious  or  too  disrupting  to 
those  being  observed,  observation  is  not  the  method  of  choice. 

Track  record  TNO 

We  have  conducted  many  studies  in  the  past  using  observational  methods.  Examples 
are: 

•  Evaluation  of  CRISISLAB,  a  crisis  management  training  simulation  used  by  the 
Netherlands  Air  Force. 

•  Observations  of  training  exercises  for  crowd  and  riot  control  units  held  by  the 
Netherlands  Military  Police  (Koninklijke  Marechaussee). 

•  Evaluation  of  the  safety  of  standing  in  football  stadiums. 

Data  collection  methods  Surveys 

Description 

When  the  primary  goal  in  research  is  not  to  describe  causal  effects,  but  rather  to 
answer  questions  about  the  distribution  of  and  relationships  among  characteristics 
of  people  as  they  exist  in  their  natural  settings,  one  can  do  survey  research.  In  doing 
survey  research,  one  collects  data  from  a  certain  population,  or  a  sample  of  that 


TNO  report  |  TNO-DV  2006  A228 


Appendix  C|  10/34 


population,  to  measure  the  relative  incidence,  distribution,  and  interrelations  of 
certain  characteristics,  attitudes  and  behaviours.  The  term  population  refers  to  the 
aggregation  of  people  to  which  the  researcher  wants  to  generalize.  Survey 
researchers  typically  gather  their  data  in  the  form  of  verbal  or  written  responses  to 
predetermined  questions.  These  questions  are  the  same  for  all  respondents 
{standardized)  and,  as  in  contrast  to  experiments,  there  is  no  manipulation  of 
variables. 

Surveys  are  often  used  to  measure  what  people  think  about  certain  topics  or  objects, 
such  as  their  work  circumstances,  government  policy,  or  the  quality  of  a  product.  In 
these  examples  the  respondents  are  asked  directly  for  their  personal  opinions  or 
attitudes.  Surveys  can  be  used  to  measure  actual  behaviour  or  factual  information. 

In  this  case,  information  is  obtained  in  an  indirect  manner  (i.e.  the  behaviour  itself 
is  not  be  measured,  only  respondents’  reports  of  their  behaviour).  When  responses 
are  indirect  the  reliability  of  the  data  may  suffer:  respondents  can  make  mistakes, 
lack  information  or  have  the  tendency  to  give  socially  desirable  answers.  For 
example,  when  the  survey  concerns  undesirable  behaviour,  such  as  driving  while 
intoxicated,  respondents  may  be  honest  about  their  real  behaviour. 

The  questionnaire 

An  essential  requirement  for  developing  good  questionnaires  is  to  have  a  clear 
conceptual  idea  of  just  what  content  is  to  be  measured.  The  structure  of  a  good 
questionnaire  is  characterized  by  three  important  elements:  1)  the  wording  of  the 
questions,  2)  the  wording  of  the  answers  and  3)  the  order  of  the  questions.  The 
wording  of  the  questions  is  perhaps  the  most  difficult  and  important  task  in 
questionnaire  construction.  Improperly  worded  questions  can  only  result  in  biased 
and  otherwise  meaningless  responses.  Attitudes,  for  example,  are  particularly 
subject  to  biases  caused  by  wording.  When  wording  questions  one  needs  to  take 
into  account  language  and  terminology,  ambiguous  questions,  suggestive  questions, 
negative  questions,  questions  that  ask  more  than  one  topic,  or  questions  that  are 
open  to  multiple  interpretations. 

When  it  comes  to  the  way  the  questions  have  to  be  answered  there  are  also  different 
possibilities.  There  are  two  basis  question  forms:  open-ended  and  closed-ended. 
Open-ended  questions  allow  the  respondent  to  answer  in  an  unconstrained  way, 
because  there  is  no  need  to  choose  between  different  alternatives  (for  example, 
asking  simply  ‘What  is  your  age?’).  Closed-ended  questions,  on  the  other  hand, 
present  two  or  more  alternatives,  and  the  respondents  select  the  choice  closest  to 
their  own  position  (e.g.,  ‘What  is  your  age?’  0-24,  25-49,  50-74).  One  can  ask 
questions  that  can  be  answered  with  a  simple  ‘yes’  or  ‘no’,  but  one  can  also  give  a 
list  with  different  alternatives.  It  is  also  possible  to  ask  respondents  to  respond  on  a 
scale  representing  a  certain  dimension.  A  common  response  scale,  often  used  in 
attitude  research,  represents  the  dimension  of  agreement  versus  disagreement. 

When  a  researcher  uses  closed-ended  questions,  he  has  to  think  carefully  about  the 
number  of  alternatives,  whether  or  not  to  present  a  middle  option,  providing  the 
response  options  ‘do  not  know’  or  ‘no  opinion’  and  the  way  in  which  the 
alternatives  are  presented.  It  is  advisable  to  always  pre-test  a  questionnaire  in  a 
small  group  of  people  before  using  it  on  a  larger  scale. 

The  order  in  which  questions  are  presented  is  important  because  questions  that 
come  earlier  in  a  questionnaire  may  affect  the  interpretation  of  questions  positioned 
later.  Furthermore,  particularly  if  a  questionnaire  is  long,  respondents  may  weary  of 
answering  questions  and  not  answer  questions  later  in  a  questionnaire  as 
thoughtfully  as  questions  presented  earlier. 
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Verbal  or  written 

Using  a  structured  questionnaire,  data  can  be  gathered  in  two  ways:  1)  with  written 
questionnaires  or  2)  with  personal  interviews.  Written  questionnaires  can  be  web- 
based  or  sent  by  mail.  Personal  interviews  can  be  done  face-to-face  or  by  telephone. 
In  both  cases  the  interviewers  need  to  be  well  trained  and  instructed.  Each  mode  has 
specific  advantages  and  disadvantages  that  the  researcher  needs  to  evaluate  in  terms 
of  their  suitability  to  the  research  question,  the  specific  population  being  studied, 
and  relative  cost. 

Survey  designs 

In  doing  survey  research,  one  always  needs  to  carefully  consider  the  choice  of 
survey  design.  This  refers  to  the  way  the  survey  is  conducted  in  terms  of  recruiting 
respondents  from  comparison  groups,  the  number  of  times  a  survey  is  conducted, 
and  so  forth.  The  most  straightforward  design  in  survey  research  seeks  to  establish 
the  incidence  and  distribution  of  characteristics  or  the  relationships  among 
characteristics.  In  studies  with  such  limited  goals,  the  only  real  concerns  are  if  the 
population  sample  is  representative  of  the  population  from  which  it  was  taken,  and 
that  the  study  accurately  measures  the  relevant  characteristics.  Such  a  design  may 
consist  of  only  one  group  of  respondents,  and  only  one  measurement  moment. 
Though  suitable  for  some  simple  research  purposes,  more  complex  research 
questions  need  more  complex  designs,  particularly  when  the  study  seeks  to 
establish  causality  between  two  or  more  variables.  For  such  studies,  various  designs 
are  available,  three  of  which  are  briefly  described. 

A  common  design  for  survey  research  that  attempts  to  explain  and  interpret  the 
relationships  between  characteristics  is  the  static-group  comparison.  This  design 
enables  the  researcher  to  study  differences  between  two  or  more  groups,  such 
between  men  and  women,  people  of  different  ages,  with  different  incomes  or  from 
different  ethnic  backgrounds.  Using  a  static-group  comparison,  we  can  better 
understand  how  different  variables  are  linked  with  group  membership,  however, 
this  design  cannot  be  used  to  establish  causality. 

One  way  to  establish  causality  is  the  time  order  criterion.  That  is,  if  the  value  of  a 
particular  variable  changes  over  time  (e.g.  income)  in  concert  with  changes  in 
another  measured  variable  (e.g.,  achieved  educational  level),  we  can  conclude  that 
changes  in  one  variable  cause  changes  in  the  other  (e.g.  higher  levels  of  education 
lead  to  increased  income).  To  meet  the  time  order  criterion  for  establishing 
causation,  we  can  use  the  panel  survey  design.  This  design  takes  into  account  time 
and  changes  over  time  by  collecting  data  at  two  or  more  points  in  time  from  the 
same  subjects  each  time. 

Sometimes  survey  researchers  are  unable  to  gather  data  by  using  the  same  subjects 
each  time.  In  that  case  the  researcher  needs  to  take  new  samples  from  the  same 
population  over  a  certain  period  of  time,  which  is  called  cross-sectional,  pseudo 
panel  design.  When  the  researcher  has  definite  ideas  about  which  variables  precede 
which  others  in  terms  of  time,  causality  can  be  established,  even  using  different 
respondent  samples. 

Assumptions 

The  sample  taken  from  the  population  of  interest  is  representative  of  that 
population. 

Respondents  are  able  to  answer  the  questions  posed  to  them.  For  example,  it  is  not 
useful  to  ask  people  what  their  blood  pressure  is,  or  what  the  length  of  their  REM 
sleep  is,  because  people  have  no  insight  into  such  processes. 
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Output 

The  data  resulting  from  surveys  are  almost  always  quantitative.  As  such  there  is  a 
variety  of  suitable  analysis  techniques  from  which  to  choose  depending  on  the 
measurement  scale  used. 

Application 

Survey  research  is  used  to  collect  data  from  a  certain  population,  or  a  sample  of  that 
population,  to  measure  the  relative  incidence,  distribution,  and  interrelations  of 
certain  characteristics,  attitudes  and  behaviour.  In  many  cases  survey  research  is 
aimed  at  establishing  whether  two  variables  co  vary,  and  under  what  conditions 
they  CO  vary.  In  general,  surveys  are  less  suitable  for  causal  explanations,  because 
variables  are  not  manipulated  in  order  to  test  causality  (though  it  can  be  done  with 
specialized  designs).  However,  questionnaires,  such  as  are  used  in  a  survey,  can  be 
a  part  of  an  experimental  research  design.  Survey  research  is  also  used  to  measure 
whether  there  are  changes  over  time,  or  between  different  groups.  Examples  of 
survey  research  include: 

The  Netherlands  army  makes  frequent  use  of  questionnaires  when  it  concerns  staff 
matters,  selection  and  training,  the  gathering  of  experiences  in  the  area  of  operation, 
attitudes  and  opinions  about  the  organization  or  new  equipment. 

Surveys  are  used  to  measure  the  effects  of  psychological  operations  in  the  area  of 
operation,  such  as  attitudes  towards  the  peacekeeping  force. 

Advantages  and  disadvantages 

Survey  research  has  many  advantages,  which  all  have  their  downsides: 

It  is  not  difficult  to  do  survey  research.  However  knowledge  about  constructing  good 
questionnaires,  sampling  and  analyzing  data  is  indispensable. 

Surveys  need  not  be  expensive.  It  depends  upon  the  number  of  respondents  (and 
whether  they  cooperate  voluntarily  or  receive  payment)  and  the  way  in  which  the  data 
are  gathered  (face-to-face,  telephone,  written  or  web-based).  For  a  simple  survey,  paper 
and  pencil  can  be  sufficient. 

Surveys  can  be  done  relatively  quickly,  since  it  does  not  take  a  lot  of  time  to  make  a 
good  questionnaire,  and  even  web-based  questionnaires  are  not  difficult  to  make 
nowadays. 

Surveys  do  not  require  a  lot  of  effort  on  the  part  the  respondents.  At  most,  respondents 
must  spend  some  time  answering  the  questions.  The  number  of  questions  that  one  can 
ask  is  not  infinite,  though.  At  some  point  respondents'  concentration  decreases,  they  get 
tired  or  distracted,  which  can  result  in  less  reliable  outcomes. 

The  resulting  data  describes  the  phenomenon  of  interest  in  quantitative  terms. 

As  mentioned  earlier,  survey  research  assumes  that  the  respondents  are  able  to  answer 
the  questions.  This  requires  a  certain  amount  of  knowledge,  insight  into  oneself  and 
honesty.  Furthermore,  it  must  be  possible  to  approach  the  respondents,  and  they  must  be 
willing  to  cooperate.  For  example,  it  will  be  very  difficult  to  do  survey  research  among 
hostile  opponents. 

Track  record  TNO 

TNO  has  a  lot  of  experience  with  survey  research.  A  few  examples  are: 

•  the  employability  of  observers; 

•  team  competencies  and  morale; 

•  stress  and  physiological  constraints; 

•  performance  and  satisfaction  in  teams; 

•  the  effects  of  psychological  operations. 
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Analysis  of  pre-existing  data 

Description 

This  data  collection  method  is,  in  fact,  a  collection  of  various  ways  to  do  archival 
research.  Specifically,  what  are  central  to  this  method,  are  the  various  sources  from 
which  input  data  for  archival  research  may  be  obtained.  Though  many  more  sources 
exist,  in  the  present  chapter,  we  address  five  of  the  most  commonly  used  sources: 
statistical  records,  written  records,  literature  study,  Meta  analysis,  and  case  study. 

Statistical  records 

This  particular  source  refers  to  the  data  available  from  bureaus  and  organizations  that 
record  statistical  information  regarding  various  phenonaena.  In  the  Netherlands,  for 
example,  the  Central  Bureau  of  Statistics  (CBS)  records  information  about  the 
demographic  make-up  of  the  population.  If  a  study  examines  the  effects  of  changes  in 
attitudes  toward  foreigners  on  immigration,  one  could  consult  the  CBS  for  information 
on  the  number  of  foreigners  (in  both  absolute  and  relative  terms)  in  the  country  across  a 
period  of  years.  In  addition,  schools,  health  agencies  and  businesses  keep  statistical 
records  of  various  types.  Particularly  useful  are  also  data  acquired  from  national  census 
taking. 

Similar  to  statistical  records  are  survey  archives.  Here,  giant  surveys  are  (repeatedly) 
conducted  in  order  to  collect  data  on  various  topics  over  a  longer  period  of  time.  These 
surveys  may  address  such  issues  as  attitudes  regarding  political  or  social  issues,  religion 
or  financial  behaviour. 

Characteristic  of  statistical  records  is  that,  first,  they  are  usually  collected  for  purposes 
other  than  the  particular  study  at  hand:  the  CBS  does  not  collect  demographic  data  in 
order  to  compare  changes  in  attitudes  with  ethnic  make-up  of  the  population.  Second, 
sometimes  the  existing  data  must  be  ‘translated’  into  indices  indicative  of  the  concepts 
that  are  being  studied.  For  example,  if  one  is  interested  in  relative  deprivation^  -  data  on 
which  is  not  directly  available  from  statistical  sources  -  one  must  look  for  secondary 
indices  of  relative  deprivation  such  as  differences  in  credit  card  debt  in  various  socio¬ 
economic  groups.  Third,  conclusions  of  studies  using  statistical  records  are  particularly 
susceptible  to  alternative  explanations. 

Written  Records 

Written  records  refer  to  documentation  created  for  a  purpose  other  than  research. 
Examples  are  speech  transcripts,  diaries,  non-fictional  literature,  or  mass 
communications.  The  analysis  of  such  information  can  yield  insights  into,  for  example, 
social  climate,  Zeitgeist,  important  themes  or  problems  in  society.  To  illustrate, 
analyzing  the  content  on  Arab  web  sites  provides  insight  into  the  way  Arab  countries 
view  their  position  in  the  world  and  how  they  interpret  Western  behaviour. 

Literature  study 

In  a  literature  study  published  information  is  collected,  analyzed  and  integrated  in  order 
to  acquire  a  relatively  complete  view  of  a  particular  topic.  Sources  are  generally 
published,  and  may  be  either  electronic  or  hard  copy.  Often,  use  is  made  of  books  or 
articles  from  scientific  sources,  the  mass  media,  the  popular  press,  web  sites  etc.  The 
goal  of  a  literature  study  is  to  acquire  an  overview  of  what  is  known  about  a  particular 
topic.  Often,  a  literature  study  is  followed  by  an  empirical  study  in  which  new  data  are 
generated,  and  that  is  intended  to  further  develop  the  existing  understanding  of  the 
phenomenon  being  studied. 


‘  Relative  deprivation  is  the  feeling  that  one  is  deprived  of  desired  resources,  which  are  available  to  others. 
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Meta  analysis 

In  much  scientific  research,  conflicting  results  may  be  found  across  studies:  results 
cannot  be  replicated,  different  research  paradigms  lead  to  different  results  etc.  In 
sciences  that  rely  on  conclusions  based  on  statistical  probabilities,  these  differences 
may  be  the  result  of  statistical  coincidence.  In  order  to  tease  apart  coincidence  and  the 
‘real’  effect,  a  Meta  analysis  can  be  used.  A  Meta  analysis,  in  this  sense,  may  be 
considered  an  extension  of  or  a  special  type  of  literature  study. 

A  Meta  analysis  is  (statistical)  method  used  to  abstract  information  about  higher-order 
effects  from  a  collection  of  primary  data  sources  (empirical  studies).  Specifically,  as 
many  studies  as  possible  examining  a  particular  concept  are  gathered.  The  relevant 
statistical  information  from  each  study  is  noted  and  mathematically  combined,  thus 
yielding  a  composite  score.  This  integrated  score  is  a  better  (more  powerful)  indicator 
of  the  relationship  between  the  examined  variables  than  the  relationships  described  in 
the  individual  studies. 

Case  study 

In  a  case  study,  one  analyses  in  detail  one  situation  in  order  to  achieve  insight  into  the 
relationships  between  relevant  variables.  In  the  case  of  the  analysis  of  existing  data,  no 
new  observations  are  carried  out  in  order  to  assess  the  case,  but  rather  existing  reports 
detailing  the  situation  of  interest  are  thoroughly  analyzed.  Though  such  a  method  may 
provide  a  detailed  view  of  the  processes  in  that  one  particular  instance,  because  case 
studies  are  ‘one-shot’  events,  the  results  of  such  a  method  lack  reliability.  Because  there 
is  no  comparison  with  other  instances,  it  is  impossible  to  conclude  that  the  identified 
relationships  are  robust  or  if  they  are  characteristic  of  only  this  one  situation.  Case 
studies  may  be  practical,  however,  in  preparation  for  a  larger  undertaking:  as  a  pilot 
study,  in  other  words.  In  this  case,  hypotheses  can  be  preliminarily  tested  to  ascertain 
whether  or  not  one  is  ‘barking  up  the  wrong  tree.’  Promising  case  study  results  can  be 
taken  as  an  indication  that  conducting  the  main  study  may  be  a  worthwhile  pursuit. 

Assumptions 

Analyses  of  existing  data  may  be  performed  if: 

•  there  are  sufficient  and  relevant  data  available  to  analyze; 

•  the  data  form  valid  indicators  of  the  concepts  being  studied. 

Output 

The  output  from  the  analysis  of  existing  data  is  dependent  upon  the  type  of  data 
analyzed  and  the  method  used  to  evaluate  or  integrate  the  data.  Some  methods  yield 
quantitative  data,  such  as  the  use  of  statistical  records  or  Meta  analyses.  Others  are 
more  descriptive  in  their  product,  such  as  a  case  study  or  the  evaluation  of  written 
records.  Even  these  qualitative  methods  may  yield  quantitative  data  if  the  data  is 
evaluated  using  quantitative  measures  (a  scorecard  method,  for  example). 

Application 

The  analysis  of  existing  data  is  generally  used  to  study  phenomena  on  a  large  scale. 
Census  data  can  be  used  to  draw  conclusions  about  the  make-up  of  a  whole  society, 
Meta  analyses  are  used  to  identify  higher-order  effects  by  combining  many  smaller 
studies  and  literature  studies  are  done  to  get  and  integrated  overview  of  what  is  known 
about  a  particular  topic.  In  addition,  existing  data  may  be  analyzed  when  studying  the 
phenomenon  is  difficult,  prohibitively  expensive  or  poses  ethical  problems.  To 
illustrate,  consider  studying  the  effects  of  non-lethal  weapons  (NEW).  Even  though  the 
effects  of  these  weapons  are  designed  to  be  less  than  lethal,  studying  them  is  tricky 
because,  for  example. 
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It  is  difficult  to  design  a  study  that  is  both  naturalistic  and  in  which  the  data  is  obtained 
under  controlled  circumstances. 

Though  the  effects  may  not  be  lethal,  they  may  cause  injury  that,  in  some  cases,  may  be 
permanent  (this  is  primarily  one  of  several  ethical  problems),  and  there  is  a  large 
amount  of  anecdotal  data  on  the  effects  of  NLW. 

Advantages 

Making  use  of  existing  data  allows  the  researcher  to  assess  the  impact  of  natural  events. 
External  validity  is  high  because  the  respondents  are  unaware  of  the  research  in  which 
they  are  participating. 

Using  existing  data  is  often  economical  because  time  and  money  are  not  needed  to 
execute  a  study. 

Useful  information  is  often  collected  en  passant  by  organizations  as  a  matter  of 
everyday  operations,  and  it  is  often  collected  repeatedly. 

Trends  over  time  are  possible. 

The  analysis  of  existing  data  is  well  suited  to  the  investigation  of  large-scale  or 
widespread  social  of  natural  phenomena. 

Disadvantages 

Internal  and  construct  validity  are  often  low. 

Relationships  between  variables  may  be  spurious  in  which  case  the  collection  of 
addition  data  is  warranted. 

Because  the  researcher  does  not  have  control  over  the  collection  of  the  data,  there  is  no 
control  over  the  particular  variables  that  are  measured,  the  presence  of  biases  and 
potential  threats  to  validity  and  reliability. 

Because  the  data  may  be  subject  to  incompleteness,  it  may  be  difficult  to  determine  the 
degree  to  which  the  data  accurately  represent  the  population  of  interest. 

Finding  relevant  information  as  well  as  information  that  can  be  used  to  rule  out 
alternative  explanations  can  be  time-consuming. 

Track  record  TNO 

Examples  of  studies  in  which  TNO  has  analyzed  existing  data  are: 

•  a  study  assessing  the  role  of  human  factors  in  Information  Operations; 

•  an  evaluation  of  the  effects  of  non-lethal  weapons  on  humans; 

•  a  review  of  the  factors  influencing  the  success  of  crisis  management  teams. 

Models  and  simulation 


Search  and  damage  assessment 

Description 

Search,  detection,  acquisition,  recognition  and  identification  are  prerequisites  for  target 
engagement.  To  detect  a  target,  a  search  is  initiated  using  sensors.  After  detection,  the 
target  is  acquired.  Acquisition  is  the  repeated  detection  of  a  target  during  several  scans 
by  the  sensors  or  over  several  seconds  of  a  continuous  look.  By  tracking  the  target  the 
processes  of  recognition  and  identification  is  started.  All  processes  (search,  detection, 
tracking,  recognition,  identification,  and  damage  assessment)  can  be  quantified  within 
success  probabilities.  The  elements  needed  for  calculating  these  probabilities  depend  on 
the  sensors  and  weapons  used,  environment,  and  of  course  the  target  itself 
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Detection  modes  and  detection  models 

Different  types  of  sensors  are  used  for  detection.  Depending  upon  the  nature  of  the 
sensor,  two  types  of  search  procedures  can  be  considered:  scanning  search  in  which  the 
search  is  conducted  using  a  succession  of  brief  glimpses  and  continuous  search  in  which 
the  target  may  be  detected  at  any  instant  of  time.  Besides  detection  models,  there  are 
search  models  to  locate  the  precise  position  of  the  target  in  the  given  area.  Examples  of 
search  models  are  exhaustive  search,  inverse  cube  law,  random  search,  and  heuristic 
search. 

Hit  probability 

After  the  target  has  been  identified  and  recognized,  either  the  shells  are  fired  from  guns, 
or  rockets,  missiles  and  torpedoes  are  launched,  or  bombs  are  dropped.  The  next  topic 
of  interest  is  to  determine  the  probability  of  hitting  the  target  and  this  depends,  besides 
Other  factors,  on  the  errors  associated  with  firing  and  launching  mechanisms.  By 
looking  at  the  probability  of  hit,  the  following  errors  are  normally  considered:  aiming 
error  and  ballistic  dispersion  error.  The  aiming  error  may  generally  arise  due  to 
improper  levelling,  alignment  of  the  weapon,  and  sensor  and  human  errors.  The  ballistic 
error  arises  mainly  due  to  meteorological  conditions  and  variations  in  the  ballistic 
characteristics  of  each  shot.  It  should  be  pointed  out  that  the  ballistic  error  is  assumed  to 
be  independent  among  rounds  within  a  fire  engagement  whereas  the  aiming  error  is 
dependent.  Due  to  these  errors  the  mean  point  of  impact  (MPI)  may  deviate  from  the 
aiming  point,  usually  the  centre  of  the  target.  The  round-to-round  dispersion  around  the 
MPI  is  used  to  determine  the  hit  probability. 

Damage  assessment 

In  a  number  of  military  OR  studies  dealing  with  artillery  guns,  air-to-surface  and 
surface-to-surface  weapons,  the  probability  or  the  proportion  of  the  area  of  a  target 
damaged  by  a  weapon  or  weapons  is  usually  required  to  be  determined.  These  studies 
broadly  pertain  to  two  categories  of  targets:  point  target  and  area  target.  Point  target  is 
one  whose  size  is  smaller  compared  to  area  damaged  by  a  single  round  of  the  attack 
weapon,  e.g.,  missile  silos,  radar  installations,  small  bridges.  In  case  the  size  of  the 
target  is  much  larger  that  the  area  damaged  by  a  single  round  of  the  attack  weapon,  it  is 
called  an  area  target,  e.g.,  city,  airfield,  harbour. 

Input 

For  the  detection  models,  input  is  necessary  about  the  probability  of  detecting  a 
target.  For  all  sensor  used  (including  visual  means)  the  probabilities  are  dependent  on 
distance  and  environment.  This  can  be  assumed  to  follow  a  distribution  (see 
assumptions),  can  be  determined  by  experts  or  experiments  or  based  on  historical  data. 
For  the  search  models  the  target  distribution  is  needed  as  input.  Again  this  can  be 
assumed  to  follow  a  distribution,  can  be  determined  by  experts  or  experiments  or  based 
on  historical  data. 

The  round-to-round  dispersion  around  the  mean  point  of  impact  is  usually  represented 
by  statistical  distributions.  This  is  used  as  input  to  determine  the  hit  probability. 

For  damage  assessment  the  probability  that  the  target  is  destroyed  and  the  damage 
function  of  the  target,  will  be  used  as  input. 

Output 

The  output  of  detection  models  are  the  probabilities  of  detecting  (a  single  target) 
within  a  certain  time.  The  output  of  search  models  is  the  location  of  the  position  of  the 
target  in  an  area.  And  of  course  the  probability  of  a  hit  is  the  output  of  the  hit 
probabilities.  The  probability  or  the  proportion  of  the  area  of  a  target  damaged  by  a 
weapon  or  weapons  is  the  output  of  the  damage  assessment. 
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Assumptions 

Concerning  search,  detection  and  damage  assessment  assumptions  are  made  for 
distributions.  With  detection  models,  assumptions  are  made  about  the  probability  of 
detecting  a  target.  There  is  often  assumed  that  the  number  of  detections  follows  a 
Poisson  distribution.  Other  possibilities  are  to  use  an  expert’s  assessment,  historical  data 
or  experiments. 

In  search  models  assumptions  are  made  about  the  target  distribution.  For  determining 
the  hit  probability,  assumptions  have  to  be  made  about  the  distribution  of  the  round-to- 
round  dispersion  and  for  damage  assessment  the  assumptions  are  concerning  the 
probability  that  the  target  is  destroyed. 

The  cumulative  probabilities  are  calculated  assuming  independence  through  total 
dependence  of  occurrences  using  assumptions. 

Advantages  and  disadvantages 

With  search  models  the  decision  maker  can  decide  how  much  equipment  is  needed  to 
accomplish  the  job  or  how  many  time  is  needed  for  the  Job.  The  disadvantage  of  the 
traditional  search  models  is  that  it  is  based  on  ‘negative  information’.  In  reality  it  is 
even  possible  that  no  target  exists.  In  that  case  the  outcome  should  be  read  as  the 
confidence  that  no  target  is  present.  For  instance  if  the  search  model  delivers  a 
probability  of  99%  after  72  hours  of  search  and  no  target  is  found  within  72  hours,  this 
should  be  interpreted  as  there  is  no  target  present  with  a  risk  of  1%. 

Application 

Search  models  are  used  when  the  aim  of  search  Lies  within  the  optimal  search  strategy. 

In  these  cases  we  talk  about  Static  Distribution  of  Effort  problems  [ref  Washburn]. 
Search  models  are  maybe  useful  in  urban  warfare.  These  methods  can  be  used  for  every 
task  where  the  yield  is  to  give  information  about  the  environment,  the  enemy  or  the 
consequences  of  firing  a  weapon. 

Track  record  TNO 

[1]  CAEn,  a  simulation  tool  for  close  combat;  Barbier;  S010259. 

[2]  IWARS. 

[3]  SMARTER,  a  simulation  tool  for  ground  based  fire  support;  Kurstjens;  SO  10067. 

[4]  SCOPE. 

Simulation  of  military  systems 

Description 

Due  to  complexity  in  defence  systems,  it  may  not  be  possible  to  perform  experiments  to 
test  and  evaluate  the  systems.  Consequently,  system  studies  are  conducted  using  models 
which  are  abstract  representations  of  one  or  more  physical  systems,  situations  or 
phenomena.  A  relative  simple  model  can  be  solved  analytically.  In  an  analytic  solution, 
the  desired  output  is  expressed  as  an  explicit  function  of  the  input  variables.  The 
requirements  of  developing  the  model  close  to  the  real  life  system  as  well  as  being 
amenable  to  simple  analytical  solution  may  not  always  be  possible.  Also,  there  are 
situations  where  an  analytical  solution  is  not  usable  for  further  analysis.  For  example, 
an  analytical  solution  may  lead  to  an  infinite  series  or  complex  integrals.  In  such  cases, 
it  is  preferable  to  use  an  experimental  approach,  namely  computer  simulation,  to 
determine  the  solution  of  the  model. 

System  simulations  can  be  governed  by  probabilistic  events.  Therefore,  one  needs  a 
mechanism  for  generating  samples  from  a  population  with  a  given  probability 
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distribution.  For  this  purpose,  a  set  of  random  numbers  is  required.  Large  number  of 
truly  random  numbers  can  be  used,  pre-generated  by  use  of  random  generators  and  store 
them  into  the  computer  memory.  The  method  of  generating  a  sample  of  values  for  the 
variable  following  a  specified  distribution  is  termed  Monte  Carlo  sampling  procedure. 

Simulation  systems  are  subdivided  in  continue  system  simulation  and  discrete  system 
simulation.  The  first  one  will  be  outlined  in  chapter  ‘Homogeneous  and  heterogeneous 
combat  models’.  In  most  military  systems,  events  do  not  occur  continuously  but  at 
isolated  points  in  time.  These  are  called  discrete  systems.  ‘Event  driven  simulation’  and 
‘time  driven  simulation’  are  two  approaches  for  discrete  system  simulation.  An 
appropriate  approach  for  simulation  depends  on  the  nature  of  the  inter-event  intervals. 
Event-driven  simulation  is  appropriate  for  unequal  or  random  intervals  between 
consecutive  events.  Contrary  to  this,  if  the  inter-event  intervals  are  equal  then  time- 
driven  simulation  approach  is  appropriate. 

Military  simulations  can  be  subdivided  into  two  groups:  weapon  system  simulations  and 
combat  simulation.  With  weapon  system  simulations  the  effectiveness  of  a  specific 
weapon  or  sensor  is  simulated.  With  combat  simulations  the  cooperation  and 
effectiveness  of  more  weapons  and  sensors  is  simulated.  Some  of  these  models 
incorporate  synergy,  but  most  don’t.  Military  simulations  can  be  classified  into  weapon 
or  sensor  system,  single  platform  (more  weapons/sensors  on  a  platform),  task  group 
(more  single-platforms  with  a  task),  mission  analysis  (more  task  groups)  and  theatre 
level  simulations  (more  missions). 

Input 

To  get  worthwhile  results  there  is  information  needed  about  the  system  which  will  be 
simulated.  When  analysing  the  damaged  area  of  a  bomb  for  example,  information  is 
needed  about  the  distribution  of  the  damage  area.  Or  for  analysing  the  survivability  of  a 
tank,  the  hit  probabilities  are  needed. 

Output 

The  results  of  the  simulation  will  provide  information  about  the  simulated  system.  This 
can  be  the  damaged  area  of  the  bomb  or  the  95%  confidence  interval  of  the  survivability 
of  a  tank.  The  results  of  the  simulation  will  always  be  output  given  the  assumptions 
which  have  been  made. 

Assumptions 

Depending  on  the  type  of  simulation  different  assumptions  can  be  made.  For  stochastic 
simulations  there  will  be  an  assumption  about  the  distribution  of  the  system. 

Advantages  and  disadvantages 

The  advantage  of  using  simulation  is  that  it  can  give  a  representation  of  a  physical 
system,  situation  or  phenomenon  when  the  model  is  too  complex  to  determine 
analytically.  Besides  that,  simulation  can  be  used  when  it  is  not  possible  or  desirable  to 
make  experiments  with  the  system  itself 

A  disadvantage  is  that  simulation  is  a  Limited  representation  of  the  reality  because 
assumptions  underlying  the  model  are  always  made. 

Application 

The  application  of  simulation  has  also  come  up  in  the  previous  paragraph.  Simulation 
can  be  used  when  determining  an  analytic  solution  is  too  complex  and  experimenting  on 
the  system  itself  is  not  desirable. 
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Track  record  TNO 

[1]  Force  Structure  Model  (FSM)  -  a  manoeuvre  combat  simulation  model; 

Overmaat;  S010356. 

[2]  SMARTER,  a  simulation  tool  for  ground  based  fire  support;  Kurstjens;  SO  10067. 

[3]  KIBOWI;  Borgers;  S030305. 

[4]  rWARS. 

[5]  Scenario  Development;  Toevank;  S020279. 

War  games 

Description 

War  gaming  can  be  described  as  [Dictionary  of  U.S.  Army  Terms  1965]: 

A  simulation  of  a  military  operation  involving  two  or  more  opposing  forces,  conducted, 
using  rules,  data  and  procedures  designed  to  depict  an  actual  or  assumed  real  life 
situation. 

Combat  simulation  and  war  gaming  are  very  closely  linked  so  much  so  that  ‘simulation’ 
is  often  used  to  refer  to  both.  However,  war  games  are  simulations  involving  human  or 
player  interactions.  Simulations  may  be  used  where  this  distinction  is  not  so  clear.  So 
war  game  involves  human  interaction  in  a  simulated  war  scenario. 

Each  war  game  has  its  purpose,  nature  and  technique  of  its  conduct.  War  games  can  be 
classified  based  on  their  purpose  as  training  games,  planning  games  and  analytical 
research  games. 


Training  war  games 

Training  games  are  essentially  meant  for  training  commanders  at  various  levels.  The 
aim  of  such  games  is  to  train  practicing  commanders  in  the  handling  of  formations  and 
units  of  various  sizes  and  the  practicing  staff  in  planning  for  various  operations  of  war 
of  for  various  contingencies.  Training  war  games  include  military  field  exercises  as  well 
as  computer-assisted  training  war  games. 

Planning  war  games 

Planning  war  games  are  played  essentially  to  test  the  effectiveness  and  reliability  of 
standard  operating  procedures  (SOP),  to  evaluate  existing  operational  plans  or  to 
develop  new  plans.  These  may  be  solo  or  two-sided  games.  In  solo  war  games,  the 
commander  makes  plans,  changes  his/her  role  and  acts  as  the  enemy  commander  and 
tests  the  plans.  This  procedure  suffers  from  the  obvious  disadvantage  that  the  enemy 
plans  are  known  to  the  player.  In  the  two-sided  war  game  the  plans  may  be  subjected  to 
scnitiny  by  an  outside  agency  or  another  commander  who  has  not  been  associated  with 
the  original  planning  at  all.  The  two-sided  game  may  be  played  where  one  of  the  teams 
is  acting  as  the  sponsor  of  the  plans  and  the  other  taking  the  role  of  the  enemy.  Planning 
war  games  can  also  be  used  when  playing  against  a  computer  which  represents  a 
‘learning’  opponent. 

Analytical  research  war  games 

The  purpose  of  these  games  is  to  test  concepts  and  doctrines,  and  to  evaluate  the 
effectiveness  of  weapons.  These  games  are  designed  as  controlled  experiments  so  that 
comparisons  can  be  made.  Situations  are  contemplated  so  that  concept  under  study 
should  form  part  of  the  game  and  with  stochastic  models  the  analysis  of  output  is  made 
using  statistical  techniques. 
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Input 

Just  as  the  input  of  simulation  of  military  systems,  the  input  will  be  information  about 
the  simulated  training,  planning,  operation  or  system  which  will  be  simulated.  This 
information  can  be  very  diverse  and  is  dependent  on  the  kind  of  war  game. 

Output 

The  results  of  the  war  games  will  be  information  about  the  situation  or  system.  For  the 
training  war  game  the  results  will  increase  the  training  capabilities  of  the  commander. 
The  planning  war  game  gives  output  about  the  effectiveness  of  a  procedure  or  plan. 
Analytic  research  war  games  will  say  more  about  the  doctrines  or  weapon  systems. 

Assumptions 

Just  like  the  input  and  output,  the  assumptions  of  war  games  can  also  vary  a  lot. 
Examples  of  assumptions  are  preconditions  about  the  tested  plan  or  operation.  Or  for 
analytic  research  war  games  the  assumptions  are  more  like  the  behaviour  of  a  group 
with  a  certain  weapon  system. 

Advantages  and  disadvantages 

The  advantage  of  using  war  games  is  that  it  can  be  used  as  a  tool  when  experimenting  in 
practise  is  not  possible  or  desirable. 

Just  like  simulation  a  disadvantage  can  be  that  war  games  are  a  limited  representation  of 
the  reality  because  there  are  always  made  assumptions  underlying  the  model. 

Application 

The  application  of  using  war  games  is  that  it  can  be  used  as  training  tool,  to  test  a 
procedure,  doctrine  or  effectiveness  of  a  weapon. 

Track  record  TNO 

[1]  CAEn,  a  simulation  tool  for  close  combat;  Barbier;  SO  10259. 

[2]  IWARS. 

[3]  KIBOWI;  Borgers;  S030305. 

[4]  Scenario  Development;  Toevank;  S020279. 

Homogeneous  and  heterogeneous  combat  models 

Description 

Homogeneous  and  heterogeneous  combat  models  model  the  relationships  of  the  mutual 
combat  strength  with  the  help  of  a  few  comparisons.  Not  only  combat  strength  will  be 
expressed  in  amount  of  weapon  systems,  but  also  aspects  like  aggregation,  time  and 
space  factors  are  taken  into  the  comparisons. 

Homogeneous  models  are  based  on  attrition  warfare.  This  means  that  in  a  fight  between 
two  armed  forces  the  number  of  losses  is  a  constant  factor  and  is  independent  of  the 
relative  strength  of  both  parties.  There  are  diverse  variants  of  this  assumption  where 
among  other  things  the  strength  of  the  opposition  influences  the  attrition  in  time  of  the 
own  units. 

On  the  basis  of  these  models  pronouncements  can  be  made  about  the  armed  force  who 
will  probably  win  the  confhct.  To  determine  this,  so  called  combat  termination  rules  are 
necessary  like  the  absolute  or  relative  wear  in  percentage  of  original  strength.  Ascertain 
from  these  rules  will  be  determined  when  the  fight  is  regarded  as  over. 

Differential  equations  can  be  formulated  where  the  strength  of  the  parties  are  dependent 
on  the  time  and  strength  of  the  opponent.  On  the  basis  of  these  equations  and  the 
combat  termination  rules  it  is  possible  to  determine  who  will  win  the  conflict  and  what 
the  strength  of  both  parties  will  be  at  the  end. 
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Empirical  studies  have  proven  that  more  factors  can  end  the  conflict  besides  the  combat 
termination  rules.  The  most  important  factors  are  in  this  case  the  tactic,  the  relative 
combat  strength  and  the  measure  of  attrition. 

In  homogeneous  models  all  used  weapons  or  combat  units  are  equal  in  the  combat.  The 
basis  of  the  heterogeneous  combat  models  is  the  homogeneous  combat  models.  The 
difference  is  that  in  heterogeneous  combat  models  several  weapon  systems  can  be 
modelled. 

The  Lanchester  model  is  an  example  of  a  homogeneous  combat  model.  Lanchester 
models  serve  as  a  fundamental  model  in  developing  theories  of  combat  and  for 
calculating  attrition  rates  in  military  operations.  The  overall  concept  is  based  on  a 
simple  mathematical  expression  where  the  attrition  rate  of  the  Blue  force  is  equal  to  the 
product  of  the  effective  firing  rate  of  the  Red  force  and  the  number  of  the  Red 
combatants  (or  weapons). 

Input 

An  important  aspect  of  combat  modelling  is  to  determine  the  rules  governing  combat 
termination.  These  rules  are  input  for  the  models  and  can  be  determined  by  experts. 

Output 

On  the  basis  of  these  models  there  can  be  said  which  party  will  probably  win  the 
conflict. 

Assumptions 

As  mentioned  in  Section  8.2  there  are  combat  termination  rules  needed  as  input  for  the 
model.  These  rules  are  also  assumptions  of  the  model. 

Advantages  and  disadvantages 

Application 

These  models  can  be  used  when  information  has  to  be  given  about  the  effectiveness  of  a 
combat,  such  as  the  amount  of  enemy  casualties. 

Track  record  TNO 

[1]  [WARS. 

[2]  Force  Structure  Model  (FSM)  -  a  manoeuvre  combat  simulation  model; 

Overmaat;  SO  1 0356. 

[3]  SCOPE. 

[4]  FOBOWl;  Borgers;  S030305. 

[5]  CAEn,  a  simulation  tool  for  close  combat;  Barbier;  S010259. 

Cost  effectiveness  analysis 

Description 

Selection  of  a  weapon  or  equipment  for  acquisition  from  available  systems  is  an 
important  class  of  problems  encountered  by  defence  decision  makers.  The  selection  of  a 
weapon  system  depends  on  its  effectiveness  and  costs.  The  evaluation  of  effectiveness 
is  usually  complex  since  it  depends  on  a  number  of  factors  affecting  the  performance  of 
the  system  For  example,  the  effectiveness  of  an  artillery  gun  system  primarily  depends 
on  its  rate  of  fire,  lethal  radius  and  accuracy.  Given  the  alternative  gun  systems,  one  gun 
system  may  be  better  than  the  other  with  respect  to  the  rate  of  fire  but  may  have  less 
lethal  radius  and  accuracy.  To  compare  the  effectiveness  of  alternative  gun  systems,  the 
analyst,  defines  a  Measure  of  Effectiveness  (MOE)  and  evaluates  it  for  available  gun 
systems. 


TNO  report  |  TNO-DV  2006  A228 


Appendix  C|  22/34 


For  estimating  system  cost,  the  systems  are  divided  into  two  categories:  systems  which 
are  to  be  developed  and  those  which  already  exist. 

For  cost  effectiveness  analysis,  the  following  approaches  are  adopted: 

Fixed  Effectiveness  Approach:  the  effectiveness  level  is  fixed  and  an  attempt  is  made  to 
determine  the  alternative  which  has  less  costs. 

Fixed  Budget  Approach:  In  this  approach  the  budget  is  fixed  and  an  attempt  is  made  to 
determine  the  alternative  which  has  maximum  effectiveness. 

Figure  of  Merit  Approach:  In  some  cases,  it  may  not  be  possible  to  fix  either  the 
effectiveness  or  the  cost.  In  such  cases  if  and  Cj,  i=l,2,...,m,  are  the  effectiveness  and 
system  cost  for  the  ith  alternative,  the  ration  Ej/Ci  gives  the  effectiveness  of  the  ith 
system  per  unit  cost  and  is  called  the  'figure  of  merit’.  The  best  choice  is  the  one  with 
the  highest  value  of  the  figure  of  merit.  Making  a  plot  of  the  effectiveness  and  costs  can 
give  a  good  insight  into  the  analysis  of  the  different  alternatives. 

Input 

This  method  will  be  applied  when  a  choice  has  to  be  made  between  a  number  of 
systems.  For  each  system  the  effectiveness  will  be  determined.  The  needed  input  will  be 
the  values  of  each  systems  by  which  the  effectiveness  can  be  calculated. 

Output 

The  output  by  using  cost  effectiveness  analyses  will  be  an  advice  which  system  is  best 
given  the  assumptions  concerning  effectiveness  and  costs.  This  means  for  the  different 
approaches  the  output  will  be: 

•  Fixed  Effectiveness  Approach:  the  costs  per  system  given  the  minimum 
effectiveness. 

•  Fixed  Budget  Approach:  the  effectiveness  per  system  given  the  maximum  costs. 

•  Figure  of  Merit  Approach:  the  effectiveness  of  each  system  per  unit  costs  (the 
figure  of  merit). 

Assumptions 

The  next  assumptions  are  made  for  this  method: 

•  It  is  assumed  that  every  criterion  identified  for  analysis  is  quantifiable  and  the 
quantification  is  unique. 

•  It  is  assumed  that  all  criteria  ca  be  related  to  one  super  criterion  (effectiveness)  and 
the  evaluation  can  be  made  on  that  basis. 

Advantages  and  disadvantages 

Comparing  to  other  analyses  is  cost  effectiveness  analysis  a  relative  simple  method 
which  gives  a  good  insight  into  the  comparison  of  different  systems  concerning  their 
costs  and  effectiveness. 

The  techniques  to  conduct  cost  effectiveness  analyses  also  have  some  fallacies,  which 
are  stated  below: 

•  Sole  criterion  fallacy:  in  cost  effectiveness  analysis  a  single  criterion  is  used  as  the 
basis  for  evaluation.  It  is  difficult  to  decide  a  single  measure  of  effectiveness  which 
considers  all  these  parameters.  Mostly  this  is  not  possible. 

•  Ratio  fallacy:  with  the  figure  of  merit  approach,  only  the  value  of  the 
effectiveness/costs  is  used  while  the  actual  magnitudes  of  the  numerator  and 
denominator  are  ignored.  It  may  be  preferable  to  use  fixed  effectiveness  approach 
of  the  fixed  budget  approach  wherever  possible. 

•  Quantification  fallacy:  Some  criteria  are  difficult  to  quantify  as  for  example  safety. 
Inter-relationship  Fallacy:  the  second  assumption  mentioned  in  Section  1.4  may  not 
be  true  because  the  underlying  criteria  are  not  independent  of  each  other. 
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•  Fallacy  in  cost  or  effectiveness  Approaches:  focing  the  budget  or  effectiveness 
inappropriately  may  lead  to  wrong  conclusions, 

•  The  number  of  missions  envisaged:  for  defence  systems  it  may  not  be  possible  to 
assess  the  number  of  missions  envisaged  during  the  planning  period.  Cost 
effectiveness  analysis  is  quite  sensitive  to  this  element. 

Application 

Cost  effectiveness  analysis  can  be  used  when  one  likes  to  make  a  choice  between 
different  systems. 

Track  record  TNO 

[1]  FEL-SALDO;  Marcel  Smit;  S020106. 

[2]  SUITCASE;  S010218. 

[3]  LCC,  Life  Cycle  Cost  Analysis,  Marcel  Smit,  S010158, 

[4]  SMP. 

Multiple  Criteria  Analysis 

Description 

Multiple  Criteria  Analysis  (MCA)  is  an  analytic  approach  for  decision-  and  decision¬ 
making  problems.  With  MCA  a  number  of  alternative  options  (concrete  or  abstract  by 
nature)  are  identified,  judged  and  compared  with  each  other.  The  identifying,  judging 
and  comparing  of  the  alternatives  is  done  with  the  help  of  a  (possibly  hierarchic) 
structured  set  of  aspects  of  judgement  (goals,  contexts,  criteria,  attributes).  These 
aspects  are  mutually  weighed  to  get  an  authorized  final  judgement  of  the  alternatives. 
Consider  for  example  the  situation  in  which  acquisition  of  weapons  is  being  planned. 
Weapons  for  acquisition  have  to  be  evaluated  on  several  criteria,  e.g.,  suitability  to  own 
forces,  maintainability  and  cost.  A  criterion  is  a  measure  of  effectiveness  and  is  the 
basis  for  evaluation.  Some  of  these  criteria  may  be  conflicting  in  the  sense  that  the  best 
suitable  weapon  can  involve  maximum  costs.  The  criteria  are  usually  not  equally 
important  and  different  stakeholders  may  have  different  opinions  about  the  importance 
of  each  criterion.  Because  of  the  subjectivity  of  the  importance  of  the  criteria  it  is 
important  to  analyse  the  effect  of  changing  weights  on  the  final  results.  This  is  called 
sensitivity  analysis. 

With  MCA  the  parties  get  insight  in  each  others  arguments  for  choosing  one  of  the 
alternatives  above  another.  There  are  two  types  of  MCA:  discrete  MCA  and  continuous 
MCA.  With  discrete  MCA  the  alternatives  are  known  and  separately  definable.  With 
continuous  MCA  the  alternatives  are  implicitly  defined  by  preconditions  (like 
mathematical  programming). 

Properties  of  MCA  are: 

•  there  is  more  than  one  criterion; 

•  there  are  conflicting  criteria  of  differing  importance  to  the  decision  makers; 

•  there  are  different  units  of  measure; 

•  (often)  there  is  no  optimal  solution  with  respect  to  conflicting  criteria; 

•  (often)  there  are  limited  alternative  solutions. 

In  the  division  Operations  Research  and  Management  of  TNO  Physics  and  Electronic 
Laboratory  a  tool  is  developed  to  support  Multiple  Criteria  Analysis.  The  tool  is  called 
TOPSYStem  There  is  also  a  system  to  support  group  discussions.  The  aim  of  the 
division’s  support  is  to  achieve  a  transparent  process  that  leads  to  argued  and  definable 
decisions  where  all  factors  of  influence  and  interest  are  taken  into  account. 

Due  to  a  lack  of  adequate  quantitative  information  about  performance  of  different 
systems,  one  may  have  to  depend  on  judgment  of  experts  rather  than  analytic  methods. 
Techniques  like  Delphi  method,  decision  matrix  approach  and  forced  decision  matrix 
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approach,  analytic  hierarchy  process  have  been  found  useful  to  analyze  this  class  of 
multiple  criteria  decision  making  problems.  These  techniques  are  briefly  discussed. 

Delphi  method 

The  Delphi  method  is  a  commonly  used  method  which  facilitates  the  use  of  the  opinion 
of  experts.  The  method  is  not  explicitly  focused  on  the  support  of  multiple  criteria 
problems. 

The  Delphi  method  is  characterized  by  three  features:  anonymity  of  the  experts, 
iteration  with  controlled  feedback  (where  there  are  more  rounds  of  questions)  and 
statistical  group  response  (where  the  median  is  taken  as  consensus).  This  method  is 
conducted  in  four  rounds. 

Decision  matrix  approach 

The  decision  matrix  approach  can  be  used  to  choose  between  different  alternatives.  The 
different  alternatives  are  judged  quantitatively  on  the  criteria.  Two  aspects  in  this 
approach  are  debatable.  The  first  aspect  is  determining  the  relevant  criteria.  The  other 
aspect  is  about  the  assignment  of  weights  to  various  criteria. 

Forced  decision  matrix  approach 

A  variant  of  the  decision  matrix  approach  is  the  forced  decision  matrix  approach.  In  this 
approach,  the  ratings  are  given  in  terms  of  0  and  I .  All  the  contractors  are  compared 
pair  wise  with  respect  to  each  criterion,  e.g.,  if  contractor  A  is  better  than  contractor  B 
in  regard  to  value  of  the  contract  we  give  a  rating  1  to  A  end  0  to  B.  The  results  of  the 
pair  wise  comparison  of  the  contractors  for  the  four  criteria  are  obtained  and  the  rating 
of  each  contractor  is  evaluated  as  rating  =  total  score/number  of  comparisons.  A 
possible  problem  can  be  the  inconsistently  of  the  pair-wise  comparisons  (A>B,  B>C  but 
not  A>C). 

Analytic  hierarchy  process 

This  technique  follows  a  similar  procedure  as  the  forced  decision  matrix  approach  but 
allows  a  wider  range  of  values  in  comparison  of  alternatives.  The  method  also  uses  pair¬ 
wise  comparison  both  for  prioritizing  alternatives  and  for  weighting  criteria  but  the 
degree  of  preference  can  be  indicated.  Further,  the  criteria  are  structured  in  a  hierarchic 
tree. 

SMARTS 

Swing  weights  into  The  Simple  Multi  attribute  Rating  Technique  (SMART)  is  a  MCA 
implementation  of  the  additive  weighting  method.  Swing  weights  have  become  a 
popular  procedure  for  assigning  weights  to  a  criterion  based  on  how  much  that  criterion 
can  influence  performance  (its  range  or  ‘swing’  in  values)  relative  to  the  criteria  with  a 
maximum  swing  in  values.  For  example,  costs  might  be  important  in  an  absolute  sense, 
but  if  all  the  alternatives  are  similar  in  costs,  then  that  criterion  will  be  less  important 
relative  to  others  being  considered.  Because  swing  weights  represent  a  ratio  of  the 
valuation  of  the  difference  in  preferences  between  any  criterion  and  the  most  influential 
criterion,  they  are  also  referred  to  in  the  literature  as  ratio-scaled  preference  judgments. 

Input 

Input  for  a  Multiple  Criteria  Analysis  is  in  the  first  place  the  set  of  alternatives  and  the 
set  of  criteria.  For  example  with  the  choice  which  car  to  buy  the  possible  cars  and  the 
criteria  (the  price,  the  fuel  consumption  etc.)  need  to  be  known.  In  the  MCA  these 
criteria  are  mutually  weighed.  The  weights  are  needed  as  input  and  can  be  determined 
by  the  decision  maker  or  by  experts. 
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Output 

The  output  of  MCA  consists  of  a  recommendation  which  alternative  or  alternatives  may 
be  better  than  the  others.  This  recommendation  is  based  on  the  critical  results  of  the 
aggregation  of  input  information,  but  also  on  a  sensitivity  analysis. 

Assumptions 

Particular  requirements  which  are  made  for  MCA  are: 

•  operable:  it  can  be  filled  in  and  used  by  reasonable  support; 

•  decomposable:  parts  of  the  problem  can  be  considered  separately; 

•  measurable:  degree  of  reaching  the  objective  should  be  able  to  be  measured  with  the 
help  of  the  attributes. 

For  the  alternatives  which  are  absorbed  the  assumptions  are: 

•  they  are  mutually  different; 

•  they  can  be  implemented; 

•  they  can  be  judged; 

•  they  contribute  to  the  objective. 

Advantages  and  disadvantages 

The  goal  and  also  an  advantage  of  MCA  is  that  the  parties  get  insight  in  each  others 
arguments  for  choosing  one  of  the  alternatives  above  another. 

The  weighing  of  the  criteria  is  on  the  basis  of  the  opinion  of  experts.  The  subjectivity  of 
the  method  can  be  seen  as  a  disadvantage. 


Application 

MCA  will  be  used  to  solve  a  problem  where  different  alternatives  are  possible.  MCA  is 
suitable  for  problems  where  the  goal  is  to  get  more  transparency  of  the  process  and  to 
get  a  recommendation  of  which  alternative  of  which  alternatives  are  better  than  others. 

Track  record  TNO 

[1]  TOPSYStem;  Wijnmalen;  S010249. 

[2]  Group  Systems  (GFR),  Wijnmalen. 

[3]  Replacement  YPR. 

Analytic  optimization  models 

Description 

There  are  many  different  optimization  models.  An  example  is  a  model  to  solve  resource 
allocation  problems.  Resource  allocation  models  optimally  use  resources  to  reach  a 
certain  goal  where  different  conditions  are  taken  into  account.  Examples  of  these 
problenas  are:  weapon  mix  problems,  weapon  deployment  problems,  weapon  target 
allocation  problems,  sortie  allocation  problems.  Besides  resource  allocation  models, 
there  are  for  instance  transportation  problems  and  assignment  problems. 

The  above  problems  can  be  solved  by  optimization  techniques.  A  couple  of  these 
techniques  are  explained  below. 

Linear  programming 

Linear  programming  is  applied  in  solving  optimization  problems  in  which  the  objective 
function  and  the  constraints  are  linear  functions.  The  application  of  linear  programming 
increased  considerably  after  the  development  of  the  simplex  method.  This  is  a  well- 
known  algorithm  using  only  function  value  information.  An  advantage  of  the  simplex 
algorithm  is  the  small  number  of  calculations  that  is  needed  to  initialize  the  algorithm. 
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Integer  programming 

By  using  linear  programming  the  solutions  may  have  some  fractional  part.  It  may  not  be 
possible  to  implement  the  solution  in  real  life  situations.  For  example,  the  solution 
obtained  by  using  linear  programming  can  obtain  4.1 1  missiles.  Since  the  weapon 
system  cannot  be  partitioned,  the  comumander  can  deploy  either  4  or  5  missiles.  This 
allocation,  however,  need  not  be  feasible  or  optimal.  To  solve  such  problems,  a 
scientific  approach  called  integer  progranaming  has  been  developed.  There  are  several 
methods  to  solve  such  problems.  Examples  are  cutting  plane  algorithms  as  well  as 
branch  and  bound  algorithms. 

Multi  objective  programming 

The  above  techniques  have  a  single  objective  function.  However,  situations  can  be 
easily  visualized  where  the  decision  maker  may  have  more  than  one  goal  to  optimize. 
For  example  a  weapon  target  allocation  problem  where  the  objective  is  to  maximize  the 
total  kill  potential  of  air  defence  weapons  against  all  attack  weapons  and  nainimize  the 
sum  of  the  shortfalls  from  the  required  kill  potential  against  the  attack  weapons.  This 
problem  combines  the  two  objectives  functions  and  since  each  of  them  is  linear,  the 
problem  reduces  to  a  multi  objective  linear  programming  problem  or  a  weighted  linear 
programming  problem. 

Dynamic  programming 

The  dynamic  programming  technique  is  based  on  the  principle  of  optimality.  This  can 
be  based  as:  an  optimal  policy  has  the  property  that  whatever  be  the  initial  state  and  the 
initial  decision,  the  remaining  decisions  constitute  an  optimal  policy  with  regard  to  the 
state  resulting  from  the  first  decision.  The  principle  of  optimality  indicates  that  there  is 
referring  to  a  decision  process  that  can  be  partitioned  into  distinct  stages  of  decisions. 
The  system  at  each  stage  will  have  a  well-defined  state  which  after  implementation  of 
the  decision  will  lead  to  some  new  state. 

Nonlinear  programming 

Situations  may  arise  where  either  the  objective  function  or  the  constraints  may  not  be 
linear.  Three  possibilities  may  arise: 

1  The  objective  function  is  linear  while  some  or  all  of  the  constraints  are  nonlinear. 

2  The  objective  function  is  nonlinear  while  all  the  constraints  are  linear. 

3  The  objective  function  is  nonlinear  and  some  or  all  of  the  constraints  are  nonlinear. 
For  solving  these  problems  nonlinear  programming  techniques  can  be  used.  Like  linear 
programming,  nonlinear  programming  is  an  optimization  technique  dealing  with 
continuous  variables. 

Input 

The  input  of  optimization  models  contains  information  about  the  problem.  For  a 
resource  allocation  problem  for  example  information  is  needed  about  the  resources  and 
the  allocations  and  the  cost,  time  or  distance  to  assign  them.  For  transportation 
problems  the  possible  means  of  transportation,  the  routes  and  the  costs  are  needed. 

Output 

The  output  of  the  optimization  model  will  be  an  analytic  optimal  solution.  The  solution 
can  be  a  solution  in  terms  of  optimal  means  of  transportation,  or  optimal  assignment. 

Assumptions 

There  are  no  general  assumptions  with  optimization  models. 
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Advantages  and  disadvantages 

In  general,  the  advantage  of  optimization  models  is  an  analytic  approach  which  results 
in  an  optimal  solution  (if  the  problem  is  solvable).  This  is  in  contrast  with  for  example 
simulation  models  or  war  games.  A  disadvantage  is  the  complexity  of  the  model  in 
particular  when  a  lot  of  variables  are  involved. 

Linear  or  multi  objective  linear  programming  is  in  general  less  complex  than  integer 
programming,  because  there  is  no  restriction  that  the  solution  has  to  be  a  round  number. 

Application 

Optimization  models  will  be  used  when  the  problem  can  be  analytically  modelled.  The 
above  discussed  optimization  techniques  don’t  have  to  be  used  separately.  The 
application  is  very  dependent  on  the  specific  problem 

Track  record  TNO 

Heuristic  optimization  models 

Description 

There  exists  a  class  of  combinatorial  problems  with  inherent  complexity  that  any 
technique  to  optimally  solve  problems  requires  too  much  computational  effort.  The 
exact  solution  of  these  problems  may  be  obtained  through  optimization  algorithms  as 
spoken  in  the  previous  chapter.  If  the  problem  is  not  amenable  to  optimization 
algorithms  due  to  large  computation  time,  another  option  is  to  go  for  solutions  which 
can  be  obtained  quickly  at  the  risk  of  sub-optimality.  These  are  called  approximation  or 
heuristic  algorithms. 

Heuristic  algorithms  are  very  efficient  in  handling  combinatorial  explosion  encountered 
in  situations  where  choices  are  sequentially  compounded,  leading  to  a  large  number  of 
alternatives.  Some  of  the  important  techniques  are  mentioned  below. 

Greedy  algorithms 

Some  optimization  problems  can  be  solved  using  a  greedy  algorithm.  A  greedy 
algorithm  builds  a  solution  iteratively.  At  each  iteration  the  algorithm  uses  a  greedy  rule 
to  make  its  choice.  Once  a  choice  is  made  the  algorithm  never  changes  its  mind  or 
looks  back  to  consider  a  different  perhaps  better  solution;  the  reason  the  algorithm  is 
called  greedy.  In  comparison  with  for  example  dynamic  programming,  discussed  in 
chapter  ‘Optimization  models’,  a  greedy  algorithm  may  fail  to  find  the  optimal  solution, 
though  it  usually  finds  a  pretty  good  one,  and  runs  much  faster  than  an  algorithm  using 
dynamic  programming. 

Simulated  annealing 

Simulated  annealing  can  be  defined  as  a  biased  random  walk  that  samples  the  objective 
function  in  the  space  of  independent  variables.  It  has  the  ability  to  migrate  trough  a 
sequence  of  local  optima  in  search  of  a  global  optimal  solution  and  to  recognize  when 
the  global  extreme  has  been  reached.  This  technique  is  a  generalized  heuristic  algorithm 
which  can  handle  large  class  of  problems  irrespective  of  the  nature  of  the  objective 
function  in  the  case  of  classical  techniques. 

Genetic  algorithms 

Genetic  algorithm  is  being  used  in  a  variety  of  applications.  It  is  a  search  and 
optimization  procedure  that  is  motivated  by  the  principle  of  natural  genetics  and  natural 
selection.  It  is  based  on  the  survival  of  fittest  principle  which  tries  to  retain  those 
species  which  are  more  suitable  under  the  prevailing  conditions.  Only  the  best  offspring 
of  the  solutions  are  retained  for  the  next  generation  of  mating;  subsequent  generations 
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proceed  in  an  evolutionary  fashion.  The  fittest  solution  which  has  been  evolved  is 
considered  to  be  an  optimal  solution. 

Artificialn  natural  networks 

With  artificial  networks  human  brain  activities  can  be  modelled  in  a  mathematical  way. 
In  this  way  diverse  neural  networks  are  developed  for  different  combinatorial  problems. 
Artificial  neural  networks  are  providing  new  approaches  to  problem  solving.  The  key  to 
utility  of  artificial  neural  networks  is  that  they  provide  a  computational  model  that  can 
be  used  to  systematize  the  process. 

Tabu  search 

Tabu  search  is  an  iterative  procedure  designed  for  the  solution  of  optimization 
problems.  The  basic  idea  of  the  method  is  to  partially  explore  the  search  space  of  all 
feasible  solutions  by  a  sequence  of  moves.  At  each  iteration,  the  move  carried  out  is  the 
most  promising  among  those  available.  A  mechanism  which  forbids  a  set  of  moves  at 
each  iteration  is  present,  aiming  to  help  the  algorithm  to  escape  from  local  (but  not 
global)  minima. 

Input 

The  input  of  heuristic  models  is  comparable  with  input  for  optimization  models. 

Output 

The  output  of  heuristic  models  is  a  suboptimal  solution  dependent  of  the  problem.  The 
output  can  for  example  be  an  alternative  that  can  be  chosen. 

Assumptions 

The  assumptions  are  also  dependent  on  the  kind  of  problem. 

Advantages  and  disadvantages 

An  advantage  of  heuristic  models  in  stead  of  analytical  ones  is  the  simplicity.  In  general 
the  computation  time  of  heuristics  is  much  less  than  of  analytic  optimization  techniques. 
An  important  disadvantage  is  the  (probable)  sub  optimality  of  the  solution. 

Application 

Mostly  heuristics  will  be  used  when  an  analytical  approach  is  too  complex. 

Track  record  TNO 
Drama  theory 
Description 

Drama  theory  addresses  problems  involving  multiple  parties  with  conflicting  objectives. 
Drama  theory  is  a  variant  of  game  theory.  In  game  theory  a  mission  is  clear  and  the 
theory  is  based  on  rational  choices  where  assumptions  and  preferences  have  to  be  stated 
in  advance  and  will  not  be  changed  in  time.  Drama  theory  takes  into  account  the  way 
people  play  games  at  different  levels  at  the  same  time  (what  they  say,  what  they  mean, 
and  what  they  do).  With  Drama  theory  players  can  show  irrational  and  emotional 
behaviour  and  its  purpose  is  to  learn  how  emotions  and  reasons  are  used  to  change 
assumptions  and  preferences.  When  communication  is  not  possible  between  the  parties, 
drama  theory  will  be  reduced  to  game  theory. 

Confrontations  are  the  principal  element  of  dramas.  A  confrontation  contains  a  number 
of  parties  involved  in  the  conflict,  each  of  whom  holds  a  particular  position.  A  peace 
support  operation  can  be  seen  as  a  linked  sequence  of  confrontations.  Confrontations 


TNO  report  |  TNO-DV  2006  A228 


Appendix  C|  29/34 


are  effectively  handled  by  a  technique  known  as  Confrontation  Analysis.  This  is  a 
method  that  applies  Drama  Theory.  It  considers  the  way  players  apply  rational- 
emotional  pressure  on  each  other  to  redefine  the  game  prior  to  it  being  played.  The 
process  of  applying  pressure  is  called  a  confrontation.  A  player  ‘wins’  a  confrontation 
by  forcing  or  persuading  others  to  redefine  the  game  so  that  it  has  an  agreed  solution  in 
line  with  the  player’s  objectives. 

With  drama  theory  the  process  can  be  seen  in  Figure  C.l.  This  process  is  used  in  the 
development  of  scenarios. 


Scene- 

setting: 

bounding 
the  situation 


Build-up: 

actors  develop 
positions  and 
fallbacks 


Climax: 

the  moment  of  truth 
generates  emotion 


♦  t  ♦  t 


Resolution: 

meeting  and  making 
agreement  ^ 


Conflict: 

heated  exchanges 

v _ ^ 

♦ 


Implementation: 

taking  action 


Figure  C.  I  Process  Drama  Theory. 

With  each  conflict  the  above  cycles  can  be  passed  through. 

The  different  phases  in  the  above  figure  are  defined  as  follows: 

•  Scene  setting:  analysis  of  the  parties,  goals,  behaviour  and  strategies.  This  is  all 
based  on  expectations  and  till  this  point  there  is  now  difference  with  game  theory. 

•  Build-up:  having  conversations  and  finding  out  each  others  strategies,  goals  and 
payoffs. 

•  Resolution:  in  this  phase  the  parties  are  working  out  the  details  where  possible 
conflicts  become  clear.  When  the  strategies  suit  each  other  and  there  is  no  conflict, 
this  phase  will  go  fast  but  when  a  conflict  arises  they  will  go  back  to  the  build-up 
phase. 

•  Climax:  if  there  is  a  conflict,  the  parties  can  influence  and  steer  each  other.  A 
change  in  strategy  or  preferences  of  one  of  the  parties  has  to  take  place  to  solve  the 
conflict. 

•  Conflict:  the  conflict  has  not  been  solved.  The  parties  break  up  and  they  have  to 
decide  if  they  actually  execute  the  threats. 

•  Implementation:  in  this  phase  the  actions  are  taken  place.  They  can’t  be  turned 
backwards. 

Advantages  and  disadvantages 

Historically,  drama  theory  has  its  roots  in  game  theory,  but  has  been  developed  to 

overcome  many  of  the  weaknesses  encountered  in  game  theoretic  treatments  of 

problems.  In  particular,  the  advantage  of  drama  theory  is  that  it  removes  the  stifling 

assumptions  of  fixed  situations  and  rationality  of  game  theory. 
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Application 

Applications  can  be  found  in  operational  analysis  of  peace  support  operations,  politics, 
psychology  and  economics. 

Track  record  TNO 

[1]  Scenario  Development;  Toevank;  S020279. 

[2]  SPECTRUM. 

[3]  http://www.dodccrp.org/events/1999/1999CCRTS/pdf_files/track_6/043howar.pdf 

[4]  http://www.ima.org.uk/conflictypapers/Bryant.pdf 

[5]  Confrontation  Analysis  by  Nigel  Howard. 

[6]  Expert  group  scenario  development:  http://intra.fel.tno.nl/sv/0015/SV- 
000062/_intranetdata/cd-rom_versiel.0/start.htm] 

Quantitative  threat  assessment:  static  and  dynamic  analyses 

Description 

This  chapter  describes  techniques  to  quantify  military  threat.  Threat  can  be  defined  as: 
any  direct  or  indirect  action  by  other  countries  which  conflicts  with  the  interests  of  a 
country.  The  threat  can  be  economic,  social,  political,  ideological  or  military. 

When  a  military  threat  exists,  a  country  will  develop  a  strategy  to  decrease  or  remove 
the  threat.  Of  course,  the  measure  of  threat  is  highly  important.  There  are  techniques 
necessary  to  quantify  the  threat. 

Static  analysis 

In  this  analysis  threats  are  compared  on  the  basis  of  number  of  weapons  and  personnel 
of  the  forces.  These  techniques  are  input  oriented  and  give  an  idea  of  the  combat 
strength- relations  hip  of  the  two  parties.  The  consequences  of  this  relationship  on  the 
course  of  the  fight  are  not  analysed  (see  dynamic  analysis).  This  has  to  be  assessed  by 
experts  using  their  experience  and  using  historical  data.  Some  static  methodologies  are 
described  below: 

Bean  count  method:  this  method  just  compares  the  amount  of  weapons  and  the  soldiers. 
Weapon  effectiveness  index  (WEI)  or  Weapon  Unit  Value  (WUV):  this  method 
determines  the  power  score  of  a  weapon  system  in  a  subjective  way,  by  weighing  the 
attributes  and  give  them  a  score. 

Potential  Anti-potential  method:  determines  the  power  score  of  a  weapon  system  in  an 
objective  way,  by  using  attrition  rates  from  simulation  models.  The  value  of  the  weapon 
system  can  be  calculated  because  it  is  proportional  to  the  measure  where  the  weapon 
can  eliminate  an  enemy  goal. 

Force  potential  using  operational  lethality  indices  (OLI):  this  method  determines  the 
force  potential  on  the  basis  of  power  scores 

Situational  Modified  Force  Strength  method  (SMFS):  this  method  determines  the  force 
strength  on  the  basis  of  the  WEI  per  weapon.  The  capacities,  anti-capacities,  deficits  in 
capacities  and  deficits  in  anti-capacities  of  certain  combat  functions  and  scenario 
aspects,  like  terrain  and  organisation  of  the  units,  are  taken  into  account. 

Dynamic  analysis 

Dynamic  models,  uses  force  strengths  or  force  potentials  in  a  simulated  scenario  to 
obtain  the  casualties  suffered  or/and  the  territory  gained  or  lost.  These  models  are 
output  oriented  and  use  mainly  the  results  of  static  methods  as  input.  With  these 
techniques  an  idea  is  given  of  the  consequences  of  the  conflict.  The  next  four  methods 
are  dynamic  analysis: 


TNO  report  |  TNO-DV  2006  A228 


Appendix  C|  31/34 


•  Quantified  Judgment  Method  of  Analysis  (QJMA):  determines  the  number  of 
wounded  persons  at  both  sides,  using  the  results  of  the  static  method  Force  Potential 
using  Operational  Lethality  Indices. 

•  Situational  Force  Scoring  (SFS):  determines  the  attrition  rates  at  both  sides,  the 
exchange  rate,  the  FLOT  movement  rate  and  the  number  of  wounded  persons  per 
combat  function,  using  the  results  of  the  static  method  SMFS. 

•  Adaptive  Dynamic  Model  (ADM);  determine  the  Force  Strength  Losses  of  two 
parties  a  day,  where  the  varying  attrition  rates  and  supplementations  are  taken  into 
account.  The  also  varying  threshold  (lower  limit  of  force  strength)  determines  when 
the  attacker  or  defender  surrenders. 

•  Arms  Race  Models:  this  method  gives  the  construction  of  the  arsenal  in  the  course 
of  time  of  two  countries  which  are  dependent  of  each  other. 

Input 

The  consequences  of  the  combat  strength  relationship  of  the  two  parties  on  the  course 
of  the  fight  are  not  analysed  with  static  analysis.  This  has  to  be  assessed  by  experts 
using  their  experience  and  using  historical  data. 

Dynamic  models  are  output  oriented  and  the  results  of  static  methods  can  be  used  as 
input. 

Output 

Static  models  are  input  oriented  and  the  results  can  be  used  for  dynamic  methods. 
Assumptions 

To  find  out  the  threats  to  a  country  the  interest  or  objectives  of  the  country  should  be 
clearly  understood. 

Disadvantage  of  static  approach 

The  basic  problem  of  static  analysis  is  that  it  does  not  provide  any  information  that 
would  help  the  analyst  or  decision  maker  in  assessing  the  threat  or  security  implications 
of  the  combat  power  of  opposite  force  postures  in  given  situations.  The  experts  need  to 
have  experience  or  well  documented  empirical  evidence  that  permits  them  to  assess  the 
static  analysis  results  by  applying  a  judgmental  model  of  combat  processes  residing  in 
their  brains. 

The  static  analysis  models  require  a  supplement  in  the  form  of  mental  appreciation  of 
the  outcomes  based  on  past  wars  and  current  force  rations.  This  is  subjective  and  the 
conclusions  may  become  controversial  in  some  cases.  The  dynamic  analysis  models  are 
too  detailed  and  adequate  information  on  weapons  and  tactics  may  not  be  built  up  in  the 
models.  Thus  they  may  not  represent  all  the  complexities  of  the  real  world  situations. 

Application 

The  quantification  of  certain  threats  gives  a  country  a  strategy  and  the  possibility  to 
assess  if  the  country  can  resist  this  threat.  Using  techniques  to  determine  the  weapon 
effectiveness,  the  force  strength  of  a  unit  and  the  waste  during  a  conflict,  are  also 
applicable  with; 

•  Combat  simulation  models:  to  model  the  weapon  effectiveness  and  the  waste. 

•  Policy  studies:  an  aid  by  determining  required  units  to  win  a  certain  conflict.  With 
this  study  the  consequents  of  a  certain  conflict  are  analysed  departing  form  a  certain 
available  arsenal  and  personnel. 

Track  record  TNO 

[1]  Defence  Requirement  Review  (DRR);  NC3A. 

[2]  ACROSS;  Klein  Baltink;  S200042. 
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Belief  networks  and  influence  diagrams 

Belief  networks  and  influence  diagrams  analyse  the  relationship  between  different 
elements.  They  are  used  to  graphically  combine  uncertain  and  dependent  events  and 
decisions.  Influence  diagrams  can  be  seen  as  an  extension  of  belief  networks. 

Belief  networks 

A  Belief  Network  is  a  directed,  acyclic  network  built  up  with  nodes  and  arcs.  An 
acyclic  network  means  that  in  the  network  there  is  no  possibility  to  return  from  a  certain 
node  to  this  node  again.  Every  node  (also  called  a  chance  node)  is  representing  an 
uncertain  event.  If  there  is  an  influence  (or  dependence)  from  one  node  to  another,  this 
will  be  described  by  an  arc  (also  called  influence  arc).  Using  these  elements,  a  Belief 
Network  can  be  extended  up  to  a  large  number  of  nodes.  Between  every  pair  of  nodes 
there  can  ex.ist  an  arrow.  Combined,  the  whole  network  is  a  representation  of  the 
uncertainty  of  the  events  in  the  network  and  represents  a  complete  distribution  of 
probability. 

A  simple  example  of  a  Belief  Network  is  given  in  Figure  C.  I .  In  this,  the  uncertainty  is 
represented  as  the  introduction  of  a  new  product:  it  will  be  successful  (‘Success')  or  not. 
Given  that  the  event  is  successful  or  not,  an  expert  produces  a  prediction  about  the 
reaction  of  the  market  (‘Prediction'),  for  example  good,  average  or  bad.  The  example 
contains  two  uncertain  events,  presented  with  an  ellipse:  ‘Success’  and  ‘Prediction’. 

The  arc  between  the  variables  ‘Success’  and  ‘Prediction’  indicates  that  these  events  are 
dependent  of  each  other.  Because  of  the  arc,  the  node  ‘Prediction’  is  called  a  child  of 
the  node  ‘Success’  and  the  node  ‘Success’  is  called  a  parent  of  the  node  ‘Prediction’. 


Figure  C.2  A  simple  Belief  Network. 

A  Belief  Network  is  used  to  illustrate  uncertain  factors,  where  the  emphasise  is  on  the 
carrying  through  of  information  known  in  the  network.  New  information  changes  the 
probability  and  can  therefore  be  used  to  determine  a  new  strategy.  Belief  Networks  can 
be  applied  qualitatively  and  quantitatively.  Qualitatively,  it  can  be  used  as  an 
illustration  of  the  interaction  of  the  underlying  problem.  Then,  it  will  be  used  as  a 
representation  of  the  causal  structure  of  a  problem  or  model.  If  a  problem  is  only 
qualitative,  it  is  called  a  Quahtative  Probabilistic  Network  (QPN)  or  Qualitative 
Certainty  Network  (QCN)  in  stead  of  a  Belief  Network.  Just  like  influence  diagrams, 
Qualitative  Probabihstic  Networks  and  Qualitative  Probabilistic  Networks  can  be  seen 
as  derivatives  of  belief  networks. 

A  Belief  Network  can  also  be  used  to  present  a  quantitative  relation  in  a  network.  The 
quantitative  relations  in  a  Belief  Network  are  represented  by  assigning  a  discrete 
distribution  of  probabiUty  to  each  node.  This  probability  describes  the  possibility  of  the 
different  outcomes  of  the  events  in  the  node  and  is  conditionally  on  the  parent  of  that 
node. 

Influence  diagrams 

Influence  Diagrams  are  used  when  the  goal  is  to  determine  the  optimal  choice  from 
different  (uncertain)  alternatives.  This  optimal  will  be  reached  if  the  alternative  has  the 
highest  benefit  or  the  highest  profit.  The  input  of  the  influence  diagrams  can  either  be 
from  experts  or  from  empirics  or  a  combination  of  these.  Just  hke  a  Belief  Network,  an 
Influence  Diagram  is  a  graphical  representation  of  a  direct,  acyclic  network.  However 
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with  Influence  Diagrams,  the  nodes  in  the  network  can,  besides  uncertain  events,  also 

present  other  types  of  variables,  namely  decisions  or  usage  value.  In  an  Influence 

Diagram  four  kinds  of  notes  can  be  found: 

•  Decision  nodes:  form:  rectangle.  The  node  can  take  different  values  which  represent 
the  possible  decisions  of  the  decision  maker, 

•  Chance  nodes:  form:  oval.  These  represent  uncertain  events  which  influence  the 
decision  nodes. 

•  Deterministic  nodes:  form:  double  oval.  The  value  of  these  nodes  is  determined 
algebraically  ascertain  from  their  (direct)  parents.  In  other  words:  if  the  value  of  the 
parents  of  a  deterministic  node  is  certainly  known,  the  value  of  the  deterministic 
node  is  also  with  certainty  known. 

•  Value/utility  nodes:  form:  diamante  or  rectangle  with  rounded  angles.  The  value  of 
the  value/utility  nodes  has  been  linked  with  the  different  possibilities  which  come 
from  the  parents.  The  meaning  of  a  value  node  can  differ  from  a  profit  to  a  measure 
of  satisfaction. 

The  form  of  the  nodes  can  differ  per  software  package.  There  are  two  kinds  of  arcs 
possible  in  an  Influence  Diagram:  influence  arcs  and  information  arcs.  The 
influence  arcs  have  the  same  meaning  as  the  arcs  in  the  Belief  Network,  namely  that 
the  event  the  arc  is  going  to  is  dependent  on  the  event  which  the  arc  comes  from. 
However,  information  arcs  are  going  to  a  decision  node  and  indicate  that  the 
decision  has  to  be  taken,  as  soon  as  the  values  of  the  direct  parents  are  known.  The 
parents  of  the  decision  nodes  are  dated.  This  is  called  Temporal  precedence’.  If 
there  are  more  decision  nodes  in  the  network,  these  have  to  be  committed  by 
information  arcs  so  that  the  different  decisions  are  made  in  a  fixed  order. 

The  example  of  Figure  C.3  can  be  extracted  as  follows: 


Figure  C,3  Example  of  an  influence  diagram. 

As  soon  as  it  is  known  what  the  market  reaction  will  be  (good,  average  or  bad)  the 
decision  has  to  be  made  if  one  will  or  will  not  invest  further  in  the  product. 

An  example  of  the  usage  of  Influence  Diagrams  is  the  ‘counter  proliferation;  of  the  US, 
i.e.  the  potency  to  resist  ‘Weapons  of  Mass  Destruction’.  This  is  a  study  of  the 
suitability  of  different  possible  Counter  proliferation  Systems  in  the  US.  An  influence 
diagram  can  be  drawn  where  the  influences  of  the  suitability  of  the  different  weapon 
systems  are  mapped. 

Input 

The  input  of  belief  networks  and  influence  diagrams  can  be  the  prediction  of  an  expert 
or  can  come  from  empirics.  In  the  previous  example  the  input  will  be  about  the  reaction 
of  the  market. 
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Output 

Belief  networks  are  used  to  map  uncertain  factors.  Output  of  a  belief  network  will 
consist  of  (conditional)  chances.  Influence  diagrams  are  used  when  the  emphasis  is  on 
determining  the  optimal  choice  between  different  uncertain  alternatives.  The  output  will 
be  which  alternative  to  choose  given  the  assumptions. 

Assumptions 

The  next  assumptions  are  made  for  belief  networks  or  influence  diagrams: 

The  arcs  are  converted  in  conditional  chances  for  a  certain  event.  This  conversion  is 
only  legitimately  if  the  network  satisfies  the  Markov  property  that  a  node  is  just 
dependent  of  his  parents.  This  Markov  property  can  be  expressed  by  the  concept  of 
conditional  independence.  Events  A  and  B  are  conditionally  independent  given  event  C, 
if 

P{A  I  C)  is  independent  of  P{B  \  C) 

In  case  of  series  of  events  without  branching,  this  definition  is  equivalent  with  a 
standard  Markov  chain,  where 

P{A\B,C)^P{^A\B) 

Where  event  A  is  a  child  of  event  B  and  event  B  is  a  child  of  event  C  (  C  B  ^  A). 
The  belief  network  or  influence  diagram  has  to  be  a  directed  network,  so  that  it  is  clear 
which  event  comes  before  another. 

The  network  has  to  be  acyclic.  Tliis  means  that  there  is  no  way  to  go  through  the 
network  from  a  node  back  to  the  same  node. 

Advantages  and  disadvantages 

Belief  Networks  and  influence  diagrams  make  it  possible  to  quantitatively  model  a 
combination  of  uncertain  factors.  The  visual  representation  can  make  aspects  of  the 
problem  very  clear.  Constructing  such  networks  will  force  the  model  maker  to  think 
about  the  dependency  between  events  in  a  structured  manner. 

One  of  the  most  important  aspects  and  big  advantage  of  a  Belief  Network  is  that 
information  known  for  an  event  can  be  directly  used  in  the  network.  This  means  that  as 
soon  as  the  outcome  of  an  event  has  become  certain,  it  is  possible  to  calculate  all 
conditional  chances  of  the  children. 

Application 

For  people  without  a  mathematical  background  (and  sometimes  even  for  people  with  a 
mathematical  background)  consideration  of  this  dependency  is  a  complex  factor.  Mostly 
a  decision  maker  tries  to  take  the  factors  which  influence  the  problem  into  account,  but 
mutual  dependence  between  factors  are  often  (mistakenly)  left  aside.  So  to  use  this 
method,  it  is  at  least  advisable  to  have  some  mathematical  and  probabilistic  experience. 
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